Original: S IMULTANEOUSLY L EARNING S TOCHASTIC AND A DVERSARIAL M ARKOV D ECISION P ROCESS WITH L INEAR F UNCTION A PPROXIMATION
Revised: S IMULTANEOUSLY L EARNING S TOCHASTIC AND A DVERSARIAL M ARKOV D ECISION P ROCESS WITH L INEAR F UNCTION A PPROXIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning (RL) has been commonly used in practice.
Revised: Reinforcement learning (RL) has been commonly used in practice.

Original: To deal with the numerous states and actions in real applications, the function approximation method has been widely employed to improve the learning efficiency, among which the linear function approximation has attracted great interest both theoretically and empirically.
Revised: To deal with the numerous states and actions in real applications, the function approximation method has been widely employed to improve the learning efficiency, among which the linear function approximation has attracted great interest both theoretically and empirically.

Original: Previous works on the linear Markov Decision Process (MDP) mainly study two settings, the stochastic setting where the reward is generated in a stochastic way and the adversarial setting where the reward can be chosen arbitrarily by an adversary.
Revised: Previous works on the linear Markov Decision Process (MDP) mainly study two settings, the stochastic setting where the reward is generated in a stochastic way and the adversarial setting where the reward can be chosen arbitrarily by an adversary.

Original: All these works treat these two environments separately.
Revised: All these works treat these two environments separately.

Original: However, the learning agents often have no idea of how rewards are generated and a wrong reward type can severely disrupt the performance of those specially designed algorithms.
Revised: However, the learning agents often have no idea of how rewards are generated and a wrong reward type can severely disrupt the performance of those specially designed algorithms.

Original: So a natural question is whether an algorithm can be derived that can efficiently learn in both environments but without knowing the reward type.
Revised: So a natural question is whether an algorithm can be derived that can efficiently learn in both environments but without knowing the reward type.

Original: In this paper, we first consider such best-of-both-worlds problem for linear MDP with the known transition.
Revised: In this paper, we first consider such best-of-both-worlds problem for linear MDP with the known transition.

Original: We propose an algorithm and prove it can simultaneously achieve O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting where K is the horizon.
Revised: We propose an algorithm and prove it can simultaneously achieve O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting where K is the horizon.

Original: To the best of our knowledge, it is the first such result for linear MDP.
Revised: To the best of our knowledge, it is the first such result for linear MDP.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) studies the problem where a learning agent interacts with the environment over time and aims to maximize its cumulative rewards in a given horizon.
Revised: Reinforcement learning (RL) studies the problem where a learning agent interacts with the environment over time and aims to maximize its cumulative rewards in a given horizon.

Original: It has a wide range of real applications including robotics (Kober et al., 2013), games (Mnih et al., 2013; Silver et al., 2016), etc.
Revised: It has a wide range of real applications including robotics (Kober et al., 2013), games (Mnih et al., 2013; Silver et al., 2016), etc.

Original: The environment dynamics are usually modeled by the Markov Decision Process (MDP) with a fixed transition function.
Revised: The environment dynamics are usually modeled by the Markov Decision Process (MDP) with a fixed transition function.

Original: We consider the general episodic MDP setting where the interactions last for several episodes and the length of each episode is fixed (Jin et al., 2018; 2020b; Luo et al., 2021; Yang et al., 2021).
Revised: We consider the general episodic MDP setting where the interactions last for several episodes and the length of each episode is fixed (Jin et al., 2018; 2020b; Luo et al., 2021; Yang et al., 2021).

Original: In each episode, the agent first observes its current state and would decide which action to take.
Revised: In each episode, the agent first observes its current state and would decide which action to take.

Original: After making the decision, it receives an instant reward and the environment will then transfer to the next state.
Revised: After making the decision, it receives an instant reward and the environment will then transfer to the next state.

Original: The cumulative reward in an episode is called the value and the objective of the agent is equivalent to minimizing the regret defined as the cumulative difference between the optimal value and its received values over episodes.
Revised: The cumulative reward in an episode is called the value and the objective of the agent is equivalent to minimizing the regret defined as the cumulative difference between the optimal value and its received values over episodes.

Original: Many previous works focus on the tabular MDP setting where the state and action space are finite and the values can be represented by a table (Azar et al., 2017; Jin et al., 2018; Chen et al., 2021;
Revised: Many previous works focus on the tabular MDP setting where the state and action space are finite and the values can be represented by a table (Azar et al., 2017; Jin et al., 2018; Chen et al., 2021;

Original: Luo et al., 2021).
Revised: Luo et al., 2021).

Original: Most of them study the stochastic setting with the stationary reward in which the reward of a state-action pair is generated from a fixed distribution (Azar et al., 2017; Jin et al., 2018; Simchowitz & Jamieson, 2019; Yang et al., 2021).
Revised: Most of them study the stochastic setting with the stationary reward in which the reward of a state-action pair is generated from a fixed distribution (Azar et al., 2017; Jin et al., 2018; Simchowitz & Jamieson, 2019; Yang et al., 2021).

Original: Since the reward may change over time in applications, some works consider the adversarial MDP where the reward can be arbitrarily generated among different episodes (Yu et al., 2009; Rosenberg & Mansour, 2019; Jin et al., 2020a; Chen et al., 2021;
Revised: Since the reward may change over time in applications, some works consider the adversarial MDP where the reward can be arbitrarily generated among different episodes (Yu et al., 2009; Rosenberg & Mansour, 2019; Jin et al., 2020a; Chen et al., 2021;

Original: Luo et al., 2021).
Revised: Luo et al., 2021).

Original: All of these works pay efforts to learn the value function table to find the optimal policy and the computation complexity highly depends on the state and action space size.
Revised: All of these works pay efforts to learn the value function table to find the optimal policy and the computation complexity highly depends on the state and action space size.

Original: However, in real applications such as the Go game, there are numerous states and the value function table is huge, which brings a great challenge to the computation complexity for traditional algorithms in the tabular case.
Revised: However, in real applications such as the Go game, there are numerous states and the value function table is huge, which brings a great challenge to the computation complexity for traditional algorithms in the tabular case.

Original: To cope with the dimensionality curse, a rich line of works employ the function approximation methods, such as the linear function and deep neural networks, to approximate the value functions or the policies to improve learning efficiency.
Revised: To cope with the dimensionality curse, a rich line of works employ the function approximation methods, such as the linear function and deep neural networks, to approximate the value functions or the policies to improve learning efficiency.

Original: These methods also achieve great success in practical scenarios such as the Atari and Go games (Mnih et al., 2013; Silver et al., 2016).
Revised: These methods also achieve great success in practical scenarios such as the Atari and Go games (Mnih et al., 2013; Silver et al., 2016).

Original: Despite their great empirical performances, it also brings a series of challenges in deriving theoretical analysis.
Revised: Despite their great empirical performances, it also brings a series of challenges in deriving theoretical analysis.

Original: To build a better theoretical understanding of these approximation methods, lots of works start from deriving regret guarantees for linear function classes.
Revised: To build a better theoretical understanding of these approximation methods, lots of works start from deriving regret guarantees for linear function classes.

Original: The linear MDP is a popular model which assumes both the transition and reward at a state-action pair are linear in the corresponding d -dimensional feature (Jin et al., 2020b;
Revised: The linear MDP is a popular model which assumes both the transition and reward at a state-action pair are linear in the corresponding d -dimensional feature (Jin et al., 2020b;

Original: He et al., 2021; Hu et al., 2022).
Revised: He et al., 2021; Hu et al., 2022).

Original: There are also mainly two types of the reward.
Revised: There are also mainly two types of the reward.

Original: For the stochastic setting, Jin et al.
Revised: For the stochastic setting, Jin et al.

Original: (2020b) provides the first efficient algorithm named Least-Square Value Iteration UCB (LSVI-UCB) and show that the its regret over K episodes can be upper bounded by O ( √ K ) .
Revised: (2020b) provides the first efficient algorithm named Least-Square Value Iteration UCB (LSVI-UCB) and show that the its regret over K episodes can be upper bounded by O ( √ K ) .

Original: To seek for a tighter result with respect to the specific problem structure, He et al.
Revised: To seek for a tighter result with respect to the specific problem structure, He et al.

Original: (2021) provide a new analysis for LSVI-UCB and show it achieves an O (poly log K ) instance-dependent regret upper bound.
Revised: (2021) provide a new analysis for LSVI-UCB and show it achieves an O (poly log K ) instance-dependent regret upper bound.

Original: The adversarial setting is much harder than the stochastic one since the reward can change arbitrarily but the agent can only observe the rewards on the experienced trajectory.
Revised: The adversarial setting is much harder than the stochastic one since the reward can change arbitrarily but the agent can only observe the rewards on the experienced trajectory.

Original: For this more challenging case, a regret upper bound of order O ( √ K ) is only obtained in the case with known transition by Neu & Olkhovskaya (2021).
Revised: For this more challenging case, a regret upper bound of order O ( √ K ) is only obtained in the case with known transition by Neu & Olkhovskaya (2021).

Original: All these works separately treat two environment types.
Revised: All these works separately treat two environment types.

Original: However, the learning agent usually has no idea of how the reward is generated.
Revised: However, the learning agent usually has no idea of how the reward is generated.

Original: And once the reward type is wrong, the specially designed algorithm for a separate setting may suffer great loss.
Revised: And once the reward type is wrong, the specially designed algorithm for a separate setting may suffer great loss.

Original: Thus deriving an algorithm that can adapt to different environment types becomes a natural solution for this problem.
Revised: Thus deriving an algorithm that can adapt to different environment types becomes a natural solution for this problem.

Original: This direction has attracted great research interest in simpler bandit (Bubeck & Slivkins, 2012; Zimmert et al., 2019; Lee et al., 2021; Kong et al., 2022) and tabular MDP settings (Jin & Luo, 2020; Jin et al., 2021b) but still remains open in linear MDP.
Revised: This direction has attracted great research interest in simpler bandit (Bubeck & Slivkins, 2012; Zimmert et al., 2019; Lee et al., 2021; Kong et al., 2022) and tabular MDP settings (Jin & Luo, 2020; Jin et al., 2021b) but still remains open in linear MDP.

Original: In this paper, we try to answer the question of deriving best-of-both-worlds (BoBW) guarantees for linear MDP.
Revised: In this paper, we try to answer the question of deriving best-of-both-worlds (BoBW) guarantees for linear MDP.

Original: Due to the challenge of learning in the adversarial setting, we also consider the known transition case.
Revised: Due to the challenge of learning in the adversarial setting, we also consider the known transition case.

Original: We propose an algorithm that continuously detects the real environment type and adjusts its strategy.
Revised: We propose an algorithm that continuously detects the real environment type and adjusts its strategy.

Original: It has been shown that our algorithm can simultaneously achieve O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting.
Revised: It has been shown that our algorithm can simultaneously achieve O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting.

Original: To the best of our knowledge, these are the first BoBW results for linear MDP.
Revised: To the best of our knowledge, these are the first BoBW results for linear MDP.

Original: It is also worth noting that our BoBW algorithm relies on an algorithm that can achieve a high-probability guarantee for the adversarial setting, which previous works fail to provide.
Revised: It is also worth noting that our BoBW algorithm relies on an algorithm that can achieve a high-probability guarantee for the adversarial setting, which previous works fail to provide.

Original: And we propose the first analysis for a high-probability regret bound in the adversarial linear MDP.
Revised: And we propose the first analysis for a high-probability regret bound in the adversarial linear MDP.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Linear MDP.
Revised: Linear MDP.

Original: Recently, deriving theoretically guaranteed algorithms for RL with linear function approximation has attracted great interests.
Revised: Recently, deriving theoretically guaranteed algorithms for RL with linear function approximation has attracted great interests.

Original: The linear MDP model is one of the most popular one.
Revised: The linear MDP model is one of the most popular one.

Original: Jin et al.
Revised: Jin et al.

Original: (2020b) develop the first efficient algorithm LSVI-UCB both in sample and computation complexity for this setting.
Revised: (2020b) develop the first efficient algorithm LSVI-UCB both in sample and computation complexity for this setting.

Original: They show that the algorithm achieves O ( √ d 3 H 3 K ) regret where d is the feature dimension and H is the length of each episode.
Revised: They show that the algorithm achieves O ( √ d 3 H 3 K ) regret where d is the feature dimension and H is the length of each episode.

Original: This result is recently improved to the optimal order O ( dH √ K ) by Hu et al.
Revised: This result is recently improved to the optimal order O ( dH √ K ) by Hu et al.

Original: with a tighter concentration analysis.
Revised: with a tighter concentration analysis.

Original: Apart from UCB, the
Revised: Apart from UCB, the

Original: TS-type algorithm has also been proposed for this setting (Zanette et al., 2020a).
Revised: TS-type algorithm has also been proposed for this setting (Zanette et al., 2020a).

Original: All these results do not consider the specific problem structure.
Revised: All these results do not consider the specific problem structure.

Original: In the stochastic setting, deriving an instance-dependent regret is more attractive to show the tighter performances of algorithms in a specific problem.
Revised: In the stochastic setting, deriving an instance-dependent regret is more attractive to show the tighter performances of algorithms in a specific problem.

Original: This type of regret has been widely studied under the tabular MDP setting (Simchowitz & Jamieson, 2019;
Revised: This type of regret has been widely studied under the tabular MDP setting (Simchowitz & Jamieson, 2019;

Original: Yang et al., 2021).
Revised: Yang et al., 2021).

Original: He et al.
Revised: He et al.

Original: (2021) is the first to provide this type of regret in linear MDP.
Revised: (2021) is the first to provide this type of regret in linear MDP.

Original: Using a different proof framework, they show that the LSVI-UCB algorithm can achieve O ( d 3 H 5 log K/ ∆) where ∆ is the minimum value gap in the episodic MDP.
Revised: Using a different proof framework, they show that the LSVI-UCB algorithm can achieve O ( d 3 H 5 log K/ ∆) where ∆ is the minimum value gap in the episodic MDP.

Original: All these works consider the stochastic setting with stationary rewards.
Revised: All these works consider the stochastic setting with stationary rewards.

Original: Neu & Olkhovskaya (2021) first attempts to analyze the more challenging adversarial environment.
Revised: Neu & Olkhovskaya (2021) first attempts to analyze the more challenging adversarial environment.

Original: They consider a simplier setting with known transition and provide an O ( √ dHK ) regret upper bound.
Revised: They consider a simplier setting with known transition and provide an O ( √ dHK ) regret upper bound.

Original: For unknown transition case, Luo et al.
Revised: For unknown transition case, Luo et al.

Original: (2021) provide an O ( d 2 / 3 H 2 K 2 / 3 ) upper bound with the help of a simulator and
Revised: (2021) provide an O ( d 2 / 3 H 2 K 2 / 3 ) upper bound with the help of a simulator and

Original: O ( d 2 H 4 K 14 / 15 ) guarantee for the general case.
Revised: O ( d 2 H 4 K 14 / 15 ) guarantee for the general case.

Original: Above all, even in the separate adversarial setting,
Revised: Above all, even in the separate adversarial setting,

Original: O ( √ K ) regret is only derived for known transition case.
Revised: O ( √ K ) regret is only derived for known transition case.

Original: We also study the known transition setting and try to provide ˜ O ( √ K ) regret in the adversarial setting while simultaneously achieving
Revised: We also study the known transition setting and try to provide ˜ O ( √ K ) regret in the adversarial setting while simultaneously achieving

Original: O (poly log K ) regret if the environment is truly stochastic.
Revised: O (poly log K ) regret if the environment is truly stochastic.

Original: Best-of-both-worlds.
Revised: Best-of-both-worlds.

Original: The question of reaching best-of-both-worlds results is first proposed by
Revised: The question of reaching best-of-both-worlds results is first proposed by

Original: Bubeck & Slivkins (2012) for bandit setting, a special case of episodic MDP with H = 1 .
Revised: Bubeck & Slivkins (2012) for bandit setting, a special case of episodic MDP with H = 1 .

Original: Their proposed algorithm assumes the setting is stochastic and continuously detects whether the assumption is satisfied.
Revised: Their proposed algorithm assumes the setting is stochastic and continuously detects whether the assumption is satisfied.

Original: Such a detection-based method is shown to achieve O (poly log K ) regret in the stochastic setting and O ( √ K ) in the adversarial setting, which is later improved by Auer & Chiang (2016).
Revised: Such a detection-based method is shown to achieve O (poly log K ) regret in the stochastic setting and O ( √ K ) in the adversarial setting, which is later improved by Auer & Chiang (2016).

Original: Similar detection-based techniques have also been adopted in more general linear bandit (Lee et al., 2021) and graph feedback (Kong et al., 2022) settings to achieve BoBW guarantees.
Revised: Similar detection-based techniques have also been adopted in more general linear bandit (Lee et al., 2021) and graph feedback (Kong et al., 2022) settings to achieve BoBW guarantees.

Original: Another line of works consider using Follow-the-Regularized-Leader (FTRL) to adapt to different environment types (Zimmert & Seldin, 2019; 2021).
Revised: Another line of works consider using Follow-the-Regularized-Leader (FTRL) to adapt to different environment types (Zimmert & Seldin, 2019; 2021).

Original: This type of algorithm is shown to be tighter than Bubeck &
Revised: This type of algorithm is shown to be tighter than Bubeck &

Original: Slivkins (2012); Auer & Chiang (2016) in the bandit setting and also attracts lots of interest in more complex problems such as combinatorial bandits (Zimmert et al., 2019; Chen et al., 2022).
Revised: Slivkins (2012); Auer & Chiang (2016) in the bandit setting and also attracts lots of interest in more complex problems such as combinatorial bandits (Zimmert et al., 2019; Chen et al., 2022).

Original: The first BoBW result in the MDP setting is provided by Jin & Luo (2020) in the tabular case.
Revised: The first BoBW result in the MDP setting is provided by Jin & Luo (2020) in the tabular case.

Original: Due to the challenge of the problem, they first study the known transition setting.
Revised: Due to the challenge of the problem, they first study the known transition setting.

Original: Their approach to achieving BoBW is the FTRL algorithm with a newly designed regularizer, which result is later improved by Jin et al.
Revised: Their approach to achieving BoBW is the FTRL algorithm with a newly designed regularizer, which result is later improved by Jin et al.

Original: (2021b)
Revised: (2021b)

Original: and also generalized to the unknown transition case.
Revised: and also generalized to the unknown transition case.

Original: To the best of our knowledge, we are the first to consider the BoBW problem in linear MDP.
Revised: To the best of our knowledge, we are the first to consider the BoBW problem in linear MDP.

Original: We also start from the known transition setting and our algorithm is based on detection.
Revised: We also start from the known transition setting and our algorithm is based on detection.

Original: RL
Revised: RL

Original: with general function approximation The linear mixture MDP is another popular RL model with linear function approximation.
Revised: with general function approximation The linear mixture MDP is another popular RL model with linear function approximation.

Original: It assumes the transition function can be approximated by a weighted average over several transition kernels.
Revised: It assumes the transition function can be approximated by a weighted average over several transition kernels.

Original: In the stochastic setting, both instance-independent (Ayoub et al., 2020; Zhou et al., 2021) and dependent regret bound (He et al., 2021) have been derived.
Revised: In the stochastic setting, both instance-independent (Ayoub et al., 2020; Zhou et al., 2021) and dependent regret bound (He et al., 2021) have been derived.

Original: And in the adversarial setting, only full information case has been studied, where the agent has access to the rewards of all state-action pairs (Cai et al., 2020;
Revised: And in the adversarial setting, only full information case has been studied, where the agent has access to the rewards of all state-action pairs (Cai et al., 2020;

Original: He et al., 2022).
Revised: He et al., 2022).

Original: Apart from linear function approximation, there is also a rich line of works considering general function classes, such as the setting with low Bellman rank (Jiang et al., 2017; Zanette et al., 2020b), low Eluder dimension (Wang et al., 2020; Kong et al., 2021) and low Bellman Eluder dimension (Jin et al., 2021a).
Revised: Apart from linear function approximation, there is also a rich line of works considering general function classes, such as the setting with low Bellman rank (Jiang et al., 2017; Zanette et al., 2020b), low Eluder dimension (Wang et al., 2020; Kong et al., 2021) and low Bellman Eluder dimension (Jin et al., 2021a).

Original:  S ETTING
Revised:  S ETTING

Original: We consider the episodic MDP setting where the agent interacts with the environment for K episodes with known transition.
Revised: We consider the episodic MDP setting where the agent interacts with the environment for K episodes with known transition.

Original: The episodic MDP can be represented by M ( S , A , H, { r k } Kk =1 , P )
Revised: The episodic MDP can be represented by M ( S , A , H, { r k } Kk =1 , P )

Original: where
Revised: where

Original: S is the state space, A is the action space, H is the length of each episode, r k = (cid:8) r k,h (cid:9) H h
Revised: S is the state space, A is the action space, H is the length of each episode, r k = (cid:8) r k,h (cid:9) H h

Original: =1 is the reward function and P =
Revised: =1 is the reward function and P =

Original: { P h } Hh =1 is the known transition probability function.
Revised: { P h } Hh =1 is the known transition probability function.

Original: Specifically, at each episode k and step h ∈
Revised: Specifically, at each episode k and step h ∈

Original: [ H ] , r k,h ( s, a ) ∈ [0 , 1] and P h ( · | s, a ) ∈ [0 , 1] |S| are the reward and transition probability at state s ∈ S by taking action a ∈ A , respectively.
Revised: [ H ] , r k,h ( s, a ) ∈ [0 , 1] and P h ( · | s, a ) ∈ [0 , 1] |S| are the reward and transition probability at state s ∈ S by taking action a ∈ A , respectively.

Original: We focus on stationary policies.
Revised: We focus on stationary policies.

Original: Denote π = { π h } Hh =1 as a policy mapping from the state space to an action distribution where π h : S → ∆ A .
Revised: Denote π = { π h } Hh =1 as a policy mapping from the state space to an action distribution where π h : S → ∆ A .

Original: For each episode k ∈ [ K ] , the agent would start from the first state s k, 1 :=
Revised: For each episode k ∈ [ K ] , the agent would start from the first state s k, 1 :=

Original: s 1 1 and determine the policy π k .
Revised: s 1 1 and determine the policy π k .

Original: Then at each step h ∈ [ H ] of episode k , it first observes the current state s k,h and then perform the action a k,h ∼ π k,h ( · | s k,h ) .
Revised: Then at each step h ∈ [ H ] of episode k , it first observes the current state s k,h and then perform the action a k,h ∼ π k,h ( · | s k,h ) .

Original: The agent would receive a random reward y k,h :=
Revised: The agent would receive a random reward y k,h :=

Original: r k,h ( s k,h , a k,h ) + ϵ k,h , where ϵ k,h is an independent zero-mean noise.
Revised: r k,h ( s k,h , a k,h ) + ϵ k,h , where ϵ k,h is an independent zero-mean noise.

Original: The environment then transfers to the next state s k,h +1 based on the transition function P ( · | s k,h , a k,h ) .
Revised: The environment then transfers to the next state s k,h +1 based on the transition function P ( · | s k,h , a k,h ) .

Original: The episode ends when the last state s k,H +1 is reached.
Revised: The episode ends when the last state s k,H +1 is reached.

Original: We focus on linear MDP with known transition where the reward functions are linear in a given feature mapping (Jin et al., 2020b;
Revised: We focus on linear MDP with known transition where the reward functions are linear in a given feature mapping (Jin et al., 2020b;

Original: He et al., 2021).
Revised: He et al., 2021).

Original: The formal definition is as follows.
Revised: The formal definition is as follows.

Original: Assumption 1.
Revised: Assumption 1.

Original: (Linear MDP with known transition)
Revised: (Linear MDP with known transition)

Original: M ( S , A , H, { r k } Kk =1 , P ) is a linear MDP with a known feature mapping ϕ : S ×
Revised: M ( S , A , H, { r k } Kk =1 , P ) is a linear MDP with a known feature mapping ϕ : S ×

Original: A → R d such that for each step h ∈
Revised: A → R d such that for each step h ∈

Original: [ H ] , there exists an unknown vector θ h where for each ( s, a ) ∈ S × A , r k,h
Revised: [ H ] , there exists an unknown vector θ h where for each ( s, a ) ∈ S × A , r k,h

Original: ( s, a ) =
Revised: ( s, a ) =

Original: (cid:10) ϕ
Revised: (cid:10) ϕ

Original: ( s, a ) , θ k,h (cid:11) .
Revised: ( s, a ) , θ k,h (cid:11) .

Original: In the stochastic setting, the reward function { r k } Kk =1 , or namely the reward parameter { θ k } Kk =1 , is fixed over different episodes k ∈
Revised: In the stochastic setting, the reward function { r k } Kk =1 , or namely the reward parameter { θ k } Kk =1 , is fixed over different episodes k ∈

Original: [ K ] .
Revised: [ K ] .

Original: And in the adversarial setting, the reward parameter { θ k } Kk = can be chosen arbitrarily by an adversary (which may be possibly dependent on previous policies).
Revised: And in the adversarial setting, the reward parameter { θ k } Kk = can be chosen arbitrarily by an adversary (which may be possibly dependent on previous policies).

Original: We evaluate the performance of a policy π by its value functions.
Revised: We evaluate the performance of a policy π by its value functions.

Original: Specifically, for any episode k and step h , denote the Q -value function Q πk,h ( s, a ) as the expected reward that will be obtained by the agent starting from ( s k,h , a k,h )
Revised: Specifically, for any episode k and step h , denote the Q -value function Q πk,h ( s, a ) as the expected reward that will be obtained by the agent starting from ( s k,h , a k,h )

Original: =
Revised: =

Original: ( s, a ) with policy π , which is formally defined as
Revised: ( s, a ) with policy π , which is formally defined as

Original: [Equation]
Revised: [Equation]

Original: Similarly, the value function V πk,h ( s ) of any state s is defined as
Revised: Similarly, the value function V πk,h ( s ) of any state s is defined as

Original: [Equation]
Revised: [Equation]

Original: In the following paper, we abuse a bit notation by using V πk :
Revised: In the following paper, we abuse a bit notation by using V πk :

Original: = V πk, 1 ( s 1 ) to represent the value of policy π at the starting state s 1 and episode k .
Revised: = V πk, 1 ( s 1 ) to represent the value of policy π at the starting state s 1 and episode k .

Original: Define ϕ π,h = E (cid:2) ϕ
Revised: Define ϕ π,h = E (cid:2) ϕ

Original: ( s h , a h ) | π (cid:3) as the expected feature vector that the policy π visits at step h .
Revised: ( s h , a h ) | π (cid:3) as the expected feature vector that the policy π visits at step h .

Original: It is worth noting that this recovers the state-action visitation probability in the tabular setting.
Revised: It is worth noting that this recovers the state-action visitation probability in the tabular setting.

Original: And according to the definition, the value function of policy π at episode k can be represented as V πk
Revised: And according to the definition, the value function of policy π at episode k can be represented as V πk

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Hh =1 (cid:10) ϕ π,h , θ k,h (cid:11) .
Revised: Hh =1 (cid:10) ϕ π,h , θ k,h (cid:11) .

Original: In this paper, we consider optimizing both the stochastic and adversarial environments within a finite policy set Π .
Revised: In this paper, we consider optimizing both the stochastic and adversarial environments within a finite policy set Π .

Original: Given a policy set Π , the learning agent would determine the policy π k ∈ Π in each episode k ∈ [ K ] .
Revised: Given a policy set Π , the learning agent would determine the policy π k ∈ Π in each episode k ∈ [ K ] .

Original: Let π ∗ ∈ arg max π (cid:80)
Revised: Let π ∗ ∈ arg max π (cid:80)

Original: Kk =

Revised: Kk =

Original: V πk be one optimal policy in Π that maximizes the cumulative value functions over K episodes, which is assumed to be unique in the stochastic setting similar to previous works in tabular MDP (Jin & Luo, 2020; Jin et al., 2021b) and bandit setting (Lee et al., 2021; Zimmert & Seldin, 2019; 2021).
Revised: V πk be one optimal policy in Π that maximizes the cumulative value functions over K episodes, which is assumed to be unique in the stochastic setting similar to previous works in tabular MDP (Jin & Luo, 2020; Jin et al., 2021b) and bandit setting (Lee et al., 2021; Zimmert & Seldin, 2019; 2021).

Original: Denote the cumulative regret compared with π ∗ ∈ Π over K episodes as
Revised: Denote the cumulative regret compared with π ∗ ∈ Π over K episodes as

Original: [Equation]
Revised: [Equation]

Original:  A LGORITHM
Revised:  A LGORITHM

Original: In this section, we propose a detection-based algorithm to optimize both stochastic and adversarial environments for linear MDP with a given policy set Π .
Revised: In this section, we propose a detection-based algorithm to optimize both stochastic and adversarial environments for linear MDP with a given policy set Π .

Original: Our algorithm is mainly inspired by the detection technique of Lee et al.
Revised: Our algorithm is mainly inspired by the detection technique of Lee et al.

Original: (2021) for BoBW in linear bandits.
Revised: (2021) for BoBW in linear bandits.

Original: At a high level, the algorithm first assumes the environment is truly adversarial and continuously detect whether it could be a stochastic one.
Revised: At a high level, the algorithm first assumes the environment is truly adversarial and continuously detect whether it could be a stochastic one.

Original: Its design relies on a new linear MDP algorithm that can return well-concentrated estimators for values of policies and also achieve sub-linear regret in the adversarial setting with high probability.
Revised: Its design relies on a new linear MDP algorithm that can return well-concentrated estimators for values of policies and also achieve sub-linear regret in the adversarial setting with high probability.

Original: Previous works on adversarial linear MDP fail to provide a high-probability guarantee and thus no existing algorithms satisfy this property.
Revised: Previous works on adversarial linear MDP fail to provide a high-probability guarantee and thus no existing algorithms satisfy this property.

Original: In Appendix D, we propose a variant of Geometric Hedge (Algorithm 4), which is initially designed for the simple bandit case (Bartlett et al., 2008), and provide a new analysis for it in the linear MDP setting.
Revised: In Appendix D, we propose a variant of Geometric Hedge (Algorithm 4), which is initially designed for the simple bandit case (Bartlett et al., 2008), and provide a new analysis for it in the linear MDP setting.

Original: We show that this algorithm satisfies the following properties and can be used to derive the BoBW results.
Revised: We show that this algorithm satisfies the following properties and can be used to derive the BoBW results.

Original: It is also worth noting that this algorithm is the first to achieve a high-probability regret guarantee for adversarial linear MDP.
Revised: It is also worth noting that this algorithm is the first to achieve a high-probability regret guarantee for adversarial linear MDP.

Original: Theorem 1.
Revised: Theorem 1.

Original: Given a policy set Π , the regret of our proposed Algorithm 4 in the adversarial setting can be upper bounded by
Revised: Given a policy set Π , the regret of our proposed Algorithm 4 in the adversarial setting can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Further, at each episode k , Algorithm 4 returns a value estimator ˆ V πk for each policy π ∈ Π .
Revised: Further, at each episode k , Algorithm 4 returns a value estimator ˆ V πk for each policy π ∈ Π .

Original: Choosing constant L 0 = 4 dH log (cid:0)
Revised: Choosing constant L 0 = 4 dH log (cid:0)

Original: | Π | /δ (cid:1) , C 1 ≥ 2 15 dH 3 log (cid:0) K | Π | /δ (cid:1) and C 2 ≥ 20 , it holds that for any k 0 ≥ L 0 and policy π ∈ Π ,
Revised: | Π | /δ (cid:1) , C 1 ≥ 2 15 dH 3 log (cid:0) K | Π | /δ (cid:1) and C 2 ≥ 20 , it holds that for any k 0 ≥ L 0 and policy π ∈ Π ,

Original: [Table]
Revised: [Table]

Original: with probability larger than 1 − δ .
Revised: with probability larger than 1 − δ .

Original: Our main BoBW algorithm is a phased algorithm and is presented in Algorithm 1.
Revised: Our main BoBW algorithm is a phased algorithm and is presented in Algorithm 1.

Original: It takes Algorithm 4 satisfying Theorem 1 with parameter L 0 and C 1 as input.
Revised: It takes Algorithm 4 satisfying Theorem 1 with parameter L 0 and C 1 as input.

Original: The first epoch is of length L 0 and the length of the following epochs would grow exponentially as Line 4.
Revised: The first epoch is of length L 0 and the length of the following epochs would grow exponentially as Line 4.

Original: During each epoch, it executes
Revised: During each epoch, it executes

Original: Algorithm 2, which we refer to as the BoBW main body (Line 3).
Revised: Algorithm 2, which we refer to as the BoBW main body (Line 3).

Original: Algorithm
Revised: Algorithm

Original: Algorithm 2 (BoBW main body) takes the Algorithm 4 with parameter C 1 and L as input.
Revised: Algorithm 2 (BoBW main body) takes the Algorithm 4 with parameter C 1 and L as input.

Original: Here L can just be regarded as the minimum number of episodes that Algorithm 4 needs to run to collect enough observations.
Revised: Here L can just be regarded as the minimum number of episodes that Algorithm 4 needs to run to collect enough observations.

Original: The algorithm first assumes the environment is adversarial and executes Algorithm 4 in at least L episodes (which we refer to as the first phase).
Revised: The algorithm first assumes the environment is adversarial and executes Algorithm 4 in at least L episodes (which we refer to as the first phase).

Original: As shown in Theorem 1, Algorithm 4 guarantees that when running for more than L episodes, the regret compared with any policy π and the distance between its estimated V value and the real V value would be no larger.
Revised: As shown in Theorem 1, Algorithm 4 guarantees that when running for more than L episodes, the regret compared with any policy π and the distance between its estimated V value and the real V value would be no larger.

Original: Based on these concentration properties, if a policy ˆ π shows consistent better performance than all of the other policies (Line 5), we have the reason to believe that the environment is truly stochastic.
Revised: Based on these concentration properties, if a policy ˆ π shows consistent better performance than all of the other policies (Line 5), we have the reason to believe that the environment is truly stochastic.

Original: Being aware of this, as shown in Line 6, Algorithm 2 would transfer to the stochastic phase (Line 9-18, which we refer to as the second phase) with the estimated V values returned by Algorithm 4.
Revised: Being aware of this, as shown in Line 6, Algorithm 2 would transfer to the stochastic phase (Line 9-18, which we refer to as the second phase) with the estimated V values returned by Algorithm 4.

Original: Since the estimated values by Algorithm 4 can well approximate the real values of policies, the exploration in the stochastic setting can be conducted by the estimated value gaps to obtain a problemdependent regret bound.
Revised: Since the estimated values by Algorithm 4 can well approximate the real values of policies, the exploration in the stochastic setting can be conducted by the estimated value gaps to obtain a problemdependent regret bound.

Original: The objective is to identify the optimal policy and maximize the collected rewards, which can be implemented by an optimization problem (Algorithm 3).
Revised: The objective is to identify the optimal policy and maximize the collected rewards, which can be implemented by an optimization problem (Algorithm 3).

Original: Taking the estimated value gap ˆ∆ as input, Algorithm 3 would return a probability distribution p ∗ over the policy set Π .
Revised: Taking the estimated value gap ˆ∆ as input, Algorithm 3 would return a probability distribution p ∗ over the policy set Π .

Original: Intuitively, p ∗ maximizes the expected values of policies while ensuring the uncertainty of all policies to be smaller than its sub-optimality gap.
Revised: Intuitively, p ∗ maximizes the expected values of policies while ensuring the uncertainty of all policies to be smaller than its sub-optimality gap.

Original: In Appendix A, we show that when ˆ∆ is estimated accurately, selecting policies based on p ∗ can reach a problem-dependent regret upper bound.
Revised: In Appendix A, we show that when ˆ∆ is estimated accurately, selecting policies based on p ∗ can reach a problem-dependent regret upper bound.

Original: Back to the main body of Algorithm 2, after computing the distribution p k based on the current estimated ˆ∆ (Line 10), it also mixes this policy distribution with a one-hot vector e ˆ π to ensure that the estimated optimal policy ˆ π can be observed for enough times and the variance of its following estimators can thus be low (Line 11).
Revised: Back to the main body of Algorithm 2, after computing the distribution p k based on the current estimated ˆ∆ (Line 10), it also mixes this policy distribution with a one-hot vector e ˆ π to ensure that the estimated optimal policy ˆ π can be observed for enough times and the variance of its following estimators can thus be low (Line 11).

Original: The algorithm then samples a policy π k according to this mixed distribution and executes it in this episode.
Revised: The algorithm then samples a policy π k according to this mixed distribution and executes it in this episode.

Original: Then based on the received rewards y k,h at each step h and the total reward Y k
Revised: Then based on the received rewards y k,h at each step h and the total reward Y k

Original: = (cid:80)
Revised: = (cid:80)

Original: Hh =1 y k,h , the value estimations of policies can be further updated.
Revised: Hh =1 y k,h , the value estimations of policies can be further updated.

Original: Due to the technical reason, here for the estimated optimal policy ˆ π and other policies π , we use different estimators.
Revised: Due to the technical reason, here for the estimated optimal policy ˆ π and other policies π , we use different estimators.

Original: Specifically, we use the importance weighted estimator to approximate the value of ˆ π .
Revised: Specifically, we use the importance weighted estimator to approximate the value of ˆ π .

Original: The reason for using mixed policy distribution ˜ p is just to ensure the low variance of this estimator.
Revised: The reason for using mixed policy distribution ˜ p is just to ensure the low variance of this estimator.

Original: And for other policies, we use the standard linear square estimators (Line 13).
Revised: And for other policies, we use the standard least square estimators (Line 13).

Original: Based on these newly estimated values of policies, the algorithm can then update the estimation for their sub-optimality gaps as Line 14.
Revised: Based on these newly estimated values of policies, the algorithm can then update the estimation for their sub-optimality gaps as Line 14.

Original: To get a tighter analysis, when computing the value gap of π , we use the traditional estimator of it by Algorithm 4 in the first k 0 episodes and the Catoni estimator for the recent k − k 0 episodes.
Revised: To get a tighter analysis, when computing the value gap of π , we use the traditional estimator of it by Algorithm 4 in the first k 0 episodes and the Catoni estimator for the recent k − k 0 episodes.

Original: Formally speaking, the Catoni estimator
Revised: Formally speaking, the Catoni estimator

Original: Catoni α (cid:0) { X 1 , X 2 , · · · X n } (cid:1) is defined as the unique root of f ( z )
Revised: Catoni α (cid:0) { X 1 , X 2 , · · · X n } (cid:1) is defined as the unique root of f ( z )

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 Φ (cid:0) α
Revised: =1 Φ (cid:0) α

Original: ( X i − z ) (cid:1) , where Φ( y )
Revised: ( X i − z ) (cid:1) , where Φ( y )

Original: = log (cid:0) 1 + y + y 2 / 2 (cid:1)
Revised: = log (cid:0) 1 + y + y 2 / 2 (cid:1)

Original: if y ≥ 0 and Φ( y ) = − log (cid:0) 1 − y + y 2 / 2 (cid:1) otherwise.
Revised: if y ≥ 0 and Φ( y ) = − log (cid:0) 1 − y + y 2 / 2 (cid:1) otherwise.

Original: The hyperparameter α πk is set as
Revised: The hyperparameter α πk is set as

Original: [Equation]
Revised: [Equation]

Original: which mainly follows Lee et al.
Revised: which mainly follows Lee et al.

Original: but with the careful consideration in MDP setting.
Revised: but with the careful consideration in MDP setting.

Original: [Equation]
Revised: [Equation]

Original: then 6:
Revised: then 6:

Original: [Equation]
Revised: [Equation]

Original: break 7: do 10:
Revised: break 7: do 10:

Original: Compute p k = OP k
Revised: Compute p k = OP k

Original: Compute ˜ p
Revised: Compute ˜ p

Original: k ( π
Revised: k ( π

Original: = π
Revised: = π

Original: Sample π k ∼ ˜ p k and execute π k 13:
Revised: Sample π k ∼ ˜ p k and execute π k 13:

Original: Receive rewards Y k
Revised: Receive rewards Y k

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Hh =1 y k,h and calculate ˆ V πk for each π ∈ Π as follows
Revised: Hh =1 y k,h and calculate ˆ V πk for each π ∈ Π as follows

Original: [Equation]
Revised: [Equation]

Original: (cid:88) h = ϕ
Revised: (cid:88) h = ϕ

Original: ⊤ π,h Σ − 1 k
Revised: ⊤ π,h Σ − 1 k

Original: [Equation]
Revised: [Equation]

Original: h ϕ π k ,h y k,h , where Σ k,h = (cid:88)
Revised: h ϕ π k ,h y k,h , where Σ k,h = (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: For each π ̸ = ˆ π , compute ˆ∆ π as
Revised: For each π ̸ = ˆ π , compute ˆ∆ π as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Rob
Revised: Rob

Original: [Equation]
Revised: [Equation]

Original: Catoni
Revised: Catoni

Original: [Equation]
Revised: [Equation]

Original: 15: if
Revised: 15: if

Original: [Equation]
Revised: [Equation]

Original: then 16:
Revised: then 16:

Original: Return k17: end if 18: end for
Revised: Return k17: end if 18: end for

Original: Is is worth noting that an adversarial setting may be disguised as stochastic scenarios and foolish the algorithm to enter in the stochastic phase.
Revised: Is is worth noting that an adversarial setting may be disguised as stochastic scenarios and fool the algorithm to enter in the stochastic phase.

Original: Thus the agent still needs to be vigilant about the possible change of the environment.
Revised: Thus the agent still needs to be vigilant about the possible change of the environment.

Original: The detection conditions (Line 15) are set for this objective.
Revised: The detection conditions (Line 15) are set for this objective.

Original: To be specific, if the estimated sub-optimality gap of a policy changes obviously compared with the original estimation by Algorithm 4 in the adversarial phase or the regret compared with ˆ π is large, the algorithm can determine that the environment may not be stochastic and would terminiate the current epoch and enter in the next epoch with parameter k 

Revised: To be specific, if the estimated sub-optimality gap of a policy changes obviously compared with the original estimation by Algorithm 4 in the adversarial phase or the regret compared with ˆ π is large, the algorithm can determine that the environment may not be stochastic and would terminiate the current epoch and enter in the next epoch with parameter k

Original: (Line 16).
Revised: (Line 16).

Original: [Equation]
Revised: [Equation]

Original: where Σ h ( p ) = (cid:80) π p π ϕ π,h ϕ ⊤ π,h .
Revised: where Σ h ( p ) = (cid:80) π p π ϕ π,h ϕ ⊤ π,h .

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we provide the theoretical guarantee and the analysis of Algorithm 1 in both stochastic and adversarial settings
Revised: In this section, we provide the theoretical guarantee and the analysis of Algorithm 1 in both stochastic and adversarial settings

Original: The first is about the stochastic setting.
Revised: The first is about the stochastic setting.

Original: Since the value function remains the same for different episodes, we simplify the notation and use V π to represent the real value of policy π ∈ Π .
Revised: Since the value function remains the same for different episodes, we simplify the notation and use V π to represent the real value of policy π ∈ Π .

Original: Before presenting the main results, we first introduce the sub-optimality gaps that will be used.
Revised: Before presenting the main results, we first introduce the sub-optimality gaps that will be used.

Original: Definition 1.
Revised: Definition 1.

Original: For each policy π ∈ Π , define ∆ π = V π ∗ − V π as the sub-optimality gap of π compared with the optimal policy π ∗ ∈ arg max π ∈ Π V π .
Revised: For each policy π ∈ Π , define ∆ π = V π ∗ − V π as the sub-optimality gap of π compared with the optimal policy π ∗ ∈ arg max π ∈ Π V π .

Original: Further let ∆ min = min π :∆ π > 0 ∆ π be the minimum non-negative value gap.
Revised: Further let ∆ min = min π :∆ π > 0 ∆ π be the minimum non-negative value gap.

Original: Theorem 2 provides a regret upper bound for Algorithm 1 in the stochastic setting.
Revised: Theorem 2 provides a regret upper bound for Algorithm 1 in the stochastic setting.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Regret bound in the stochastic setting) With probability at least 1 − δ , Algorithmguarantees that
Revised: (Regret bound in the stochastic setting) With probability at least 1 − δ , Algorithmguarantees that

Original: [Equation]
Revised: [Equation]

Original: And if the environment is adversarial, the regret of Algorithm 1 can be upper bounded as Theorem 3.
Revised: And if the environment is adversarial, the regret of Algorithm 1 can be upper bounded as Theorem 3.

Original: Theorem 3.
Revised: Theorem 3.

Original: (Regret bound in the adversarial setting)
Revised: (Regret bound in the adversarial setting)

Original: With probability at least 1 − δ , Algorithmguarantees that
Revised: With probability at least 1 − δ , Algorithmguarantees that

Original: [Equation]
Revised: [Equation]

Original: Due to the space limit, the full proof of these two theorems are deferred to Appendix B and C. We will provide a proof sketch for them in later sections.
Revised: Due to the space limit, the full proof of these two theorems are deferred to Appendix B and C. We will provide a proof sketch for them in later sections.

Original: Technique challenge and novelty.
Revised: Technique challenge and novelty.

Original: There are mainly two types of algorithms to deal with the
Revised: There are mainly two types of algorithms to deal with the

Original: BoBW problem: the switch-based method which actively detects the environment type, e.g., Bubeck & Slivkins (2012), and the FTRL-based method which adapts to different environments, e.g. Zimmert & Seldin (2019).
Revised: BoBW problem: the switch-based method which actively detects the environment type, e.g., Bubeck & Slivkins (2012), and the FTRL-based method which adapts to different environments, e.g. Zimmert & Seldin (2019).

Original: The approach in Bubeck & Slivkins (2012) first assumes the setting to be stochastic and would detect whether a policy’s value has changed.
Revised: The approach in Bubeck & Slivkins (2012) first assumes the setting to be stochastic and would detect whether a policy’s value has changed.

Original: Such an approach in our setting brings an
Revised: Such an approach in our setting brings an

Original: O ( (cid:112) | Π | ) dependence in the regret for adversarial setting which is not idealistic as the policy set size can be large.
Revised: O ( (cid:112) | Π | ) dependence in the regret for adversarial setting which is not idealistic as the policy set size can be large.

Original: And the success of FTRL for BoBW mainly relies on a self-bounding inequality that bounds the regret by the chosen probabilities of policies.
Revised: And the success of FTRL for BoBW mainly relies on a self-bounding inequality that bounds the regret by the chosen probabilities of policies.

Original: But such a technique is challenging with linear structure.
Revised: But such a technique is challenging with linear structure.

Original: As discussed by Lee et al.
Revised: As discussed by Lee et al.

Original: (2021), even for the single-state linear bandit setting, connecting FTRL with OP is hard.
Revised: (2021), even for the single-state linear bandit setting, connecting FTRL with OP is hard.

Original: Our approach relies on a new observation that the value of each policy can be written as the inner product between the expected state-action visitation feature ϕ π and the unknown reward feature θ .
Revised: Our approach relies on a new observation that the value of each policy can be written as the inner product between the expected state-action visitation feature ϕ π and the unknown reward feature θ .

Original: From this view, we are able to reduce the problem to linear optimization and existing techniques for linear optimization can be used.
Revised: From this view, we are able to reduce the problem to linear optimization and existing techniques for linear optimization can be used.

Original: To the best of our knowledge, we are the first to introduce this type of linear optimization for the regret minimization problem in MDP and such reduction may be of independent interest.
Revised: To the best of our knowledge, we are the first to introduce this type of linear optimization for the regret minimization problem in MDP and such reduction may be of independent interest.

Original: Relationship between our ∆ π and the gap min in He et al.
Revised: Relationship between our ∆ π and the gap min in He et al.

Original: To compare our ∆ π with gap min in He at al.
Revised: To compare our ∆ π with gap min in He at al.

Original: (2021), we assume the optimal policy π ∗ ∈ Π is just the global optimal policy.
Revised: (2021), we assume the optimal policy π ∗ ∈ Π is just the global optimal policy.

Original: Recall that gap min is defined as min h,s,a (cid:8) gap h
Revised: Recall that gap min is defined as min h,s,a (cid:8) gap h

Original: ( s, a ) :
Revised: ( s, a ) :

Original: gap h ( s, a ) > 0 (cid:9) where gap h ( s, a ) =
Revised: gap h ( s, a ) > 0 (cid:9) where gap h ( s, a ) =

Original: V ∗ h ( s ) −
Revised: V ∗ h ( s ) −

Original: Q ∗ h ( s, a ) .
Revised: Q ∗ h ( s, a ) .

Original: In general, ∆ π can be decomposed as ∆ π = V ∗ 1 ( s 1 ) − V π 1 ( s 1 ) =
Revised: In general, ∆ π can be decomposed as ∆ π = V ∗ 1 ( s 1 ) − V π 1 ( s 1 ) =

Original: V ∗ 1 ( s 1 ) − Q π 1 ( s 1 , a ′ ) ≥
Revised: V ∗ 1 ( s 1 ) − Q π 1 ( s 1 , a ′ ) ≥

Original: V ∗ 1 ( s 1 ) − Q ∗ 1 ( s 1 , a ′ ) =
Revised: V ∗ 1 ( s 1 ) − Q ∗ 1 ( s 1 , a ′ ) =

Original: gap 1 ( s 1 , a ′ ) where the first equality follows He et al.
Revised: gap 1 ( s 1 , a ′ ) where the first equality follows He et al.

Original: (2021, Eq.
Revised: (2021, Eq.

Original: (B.2)) and p πh ( s, a ) is the visitation probability of state-action ( s, a ) at step h by following π .
Revised: (B.2)) and p πh ( s, a ) is the visitation probability of state-action ( s, a ) at step h by following π .

Original: This shows that ∆ π ≥ p π gap min in the worst case where p π is the minimum none-zero visitation probability of policy π at some state-action pair.
Revised: This shows that ∆ π ≥ p π gap min in the worst case where p π is the minimum none-zero visitation probability of policy π at some state-action pair.

Original: And there are also cases where our defined ∆ π is larger than that in He et al.
Revised: And there are also cases where our defined ∆ π is larger than that in He et al.

Original: When both the policy and transition are deterministic (Ortner, 2010; Tranos & Proutiere, 2021; Dann et al., 2021;
Revised: When both the policy and transition are deterministic (Ortner, 2010; Tranos & Proutiere, 2021; Dann et al., 2021;

Original: Tirinzoni et al., 2022), we have ∆ π = (cid:80)
Revised: Tirinzoni et al., 2022), we have ∆ π = (cid:80)

Original: Hh =1 gap h ( s h , a h ) ≥ gap min .
Revised: Hh =1 gap h ( s h , a h ) ≥ gap min .

Original: And in the stochastic transition case, if all sub-optimal policies happen to not select the optimal action in arg max a Q ∗ 1 ( s 1 , a ) , ∆ π = V
Revised: And in the stochastic transition case, if all sub-optimal policies happen to not select the optimal action in arg max a Q ∗ 1 ( s 1 , a ) , ∆ π = V

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ≥ gap min , where a ′ is the action selected by π at the first step and the last inequality is due to gap 1 ( s 1 , a ′ ) > 0 .
Revised: ≥ gap min , where a ′ is the action selected by π at the first step and the last inequality is due to gap 1 ( s 1 , a ′ ) > 0 .

Original: In the above two cases, our our sub-optimality gap is larger than previously defined gap and our dependence on the gap is better.
Revised: In the above two cases, our our sub-optimality gap is larger than previously defined gap and our dependence on the gap is better.

Original: To the best of our knowledge, Algorithm 1 is the first that can simultaneously achieve O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting for linear MDP problem.
Revised: To the best of our knowledge, Algorithm 1 is the first that can simultaneously achieve O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting for linear MDP problem.

Original: It is also worth noting that previous works on the separate adversarial setting only provide the upper bound for the expected regret (Neu & Olkhovskaya, 2021), and we are the first to provide a high-probability guarantee.
Revised: It is also worth noting that previous works on the separate adversarial setting only provide the upper bound for the expected regret (Neu & Olkhovskaya, 2021), and we are the first to provide a high-probability guarantee.

Original: 5. R EGRET A NALYSIS IN THE S TOCHASTIC S ETTING
Revised: 5. R EGRET A NALYSIS IN THE S TOCHASTIC S ETTING

Original: In the stochastic setting, we consider the regret in two phases of each epoch separately.
Revised: In the stochastic setting, we consider the regret in two phases of each epoch separately.

Original: We first show that, the first phase (which we call as the adversarial phase, Line 3-8 in Algorithm 2) will terminate after k 0 episodes where k 0 ∈ (cid:2) 64 f K C 1 / ∆ 2min , 900 f K C 1 / ∆ 2min (cid:3) with high probability, and the optimal policy π ∗ ∈ Π can be identified.
Revised: We first show that, the first phase (which we call as the adversarial phase, Line 3-8 in Algorithm 2) will terminate after k 0 episodes where k 0 ∈ (cid:2) 64 f K C 1 / ∆ 2min , 900 f K C 1 / ∆ 2min (cid:3) with high probability, and the optimal policy π ∗ ∈ Π can be identified.

Original: Lemma 1 summarizes the formal claims.
Revised: Lemma 1 summarizes the formal claims.

Original: Lemma 1.
Revised: Lemma 1.

Original: In the stochastic setting, in each epoch, the following 4 claims hold.
Revised: In the stochastic setting, in each epoch, the following 4 claims hold.

Original: With probability at least 1 − 4 δ , k 0 ≤ max (cid:110) 900 f K C 1 ∆ 2min , L (cid:111) .
Revised: With probability at least 1 − 4 δ , k 0 ≤ max (cid:110) 900 f K C 1 ∆ 2min , L (cid:111) .

Original: With probability at least 1 − δ , ˆ π = π ∗ .
Revised: With probability at least 1 − δ , ˆ π = π ∗ .

Original: With probability at least 1 − 2 δ , k 0 ≥ 64 f K C 1 ∆ 2min .
Revised: With probability at least 1 − 2 δ , k 0 ≥ 64 f K C 1 ∆ 2min .

Original: With probability at least 1 − 3 δ , ˆ∆ π ∈ [0 .
Revised: With probability at least 1 − 3 δ , ˆ∆ π ∈ [0 .

Original: 7∆ π , 1 .
Revised: 7∆ π , 1 .

Original: 3∆ π ] , ∀ π ̸ = π ∗ .
Revised: 3∆ π ] , ∀ π ̸ = π ∗ .

Original: The detailed proof of Lemma 1 is deferred to Appendix B. We next will give a proof sketch of
Revised: The detailed proof of Lemma 1 is deferred to Appendix B. We next will give a proof sketch of

Original: Theorem 2 based on the results of Lemma 1.
Revised: Theorem 2 based on the results of Lemma 1.

Original: According to claim 1 in Lemma 1, we know that k 

Revised: According to claim 1 in Lemma 1, we know that k

Original: =
Revised: =

Original: O (cid:0) f K C 1 / ∆ 2min (cid:1) , so we can bound the regret in the first phase using the guarantees of
Revised: O (cid:0) f K C 1 / ∆ 2min (cid:1) , so we can bound the regret in the first phase using the guarantees of

Original: Algorithm 4 in Theorem 1 by √
Revised: Algorithm 4 in Theorem 1 by √

Original: [Equation]
Revised: [Equation]

Original: And after k 0 episodes, the algorithm would transfer to the second phase, which we call as the stochastic phase (Line 9-18 in Algorithm 2).
Revised: And after k 0 episodes, the algorithm would transfer to the second phase, which we call as the stochastic phase (Line 9-18 in Algorithm 2).

Original: If the environment is truly stochastic, the values of all policies would remain stationary and the detection condition (Line 15 in Algorithm 2) would never be satisfied.
Revised: If the environment is truly stochastic, the values of all policies would remain stationary and the detection condition (Line 15 in Algorithm 2) would never be satisfied.

Original: Thus the stochastic phase will not end (proved in Lemma 8 in Appendix B).
Revised: Thus the stochastic phase will not end (proved in Lemma 8 in Appendix B).

Original: As for the regret suffered in this phase, we can analyze it using the properties of OP .
Revised: As for the regret suffered in this phase, we can analyze it using the properties of OP .

Original: According to claim 4 in Lemma 1, the estimated sub-optimality gap is close to the real sub-optimality gap as ˆ∆ π ∈ (cid:104) ∆ π / √ 3 , √ 3∆ π (cid:105) .
Revised: According to claim 4 in Lemma 1, the estimated sub-optimality gap is close to the real sub-optimality gap as ˆ∆ π ∈ (cid:104) ∆ π / √ 3 , √ 3∆ π (cid:105) .

Original: Thus performing policies based on the solution of OP with ˆ∆ can reach the real instance optimality.
Revised: Thus performing policies based on the solution of OP with ˆ∆ can reach the real instance optimality.

Original: Specifically, we can first bridge the gap between the expected regret and the regret that occurred using
Revised: Specifically, we can first bridge the gap between the expected regret and the regret that occurred using

Original: Freedman inequality (Lemma 12).
Revised: Freedman inequality (Lemma 12).

Original: [Equation]
Revised: [Equation]

Original: Recall that ˜ p is computed based on OP under ˆ∆ , which is close to the real sub-optimality gap ∆ .
Revised: Recall that ˜ p is computed based on OP under ˆ∆ , which is close to the real sub-optimality gap ∆ .

Original: The regret occurred in phase 2 in episodes k larger than a problem dependent constant M =
Revised: The regret occurred in phase 2 in episodes k larger than a problem dependent constant M =

Original: O (cid:0) dHβ K / ∆ 2min (cid:1) , which is the dominating part in the expected regret, can be bounded using Lemma 7 with r = 3 :
Revised: O (cid:0) dHβ K / ∆ 2min (cid:1) , which is the dominating part in the expected regret, can be bounded using Lemma 7 with r = 3 :

Original: [Equation]
Revised: [Equation]

Original: Above all, we can conclude that with probability at least 1 − δ , the regret can be upper bounde as
Revised: Above all, we can conclude that with probability at least 1 − δ , the regret can be upper bounde as

Original: [Equation]
Revised: [Equation]

Original: 5. R EGRET A NALYSIS IN THE A DVERSARIAL S ETTING
Revised: 5. R EGRET A NALYSIS IN THE A DVERSARIAL S ETTING

Original: In the adversarial setting, the regret in the first phase can be guaranteed with the property of Algorithm 4 in Theorem 1.
Revised: In the adversarial setting, the regret in the first phase can be guaranteed with the property of Algorithm 4 in Theorem 1.

Original: Here we will mainly analyze the second phase.
Revised: Here we will mainly analyze the second phase.

Original: We first show that in the second phase of each epoch, the returned policy ˆ π is actually the optimal policy in Π .
Revised: We first show that in the second phase of each epoch, the returned policy ˆ π is actually the optimal policy in Π .

Original: Lemma 2.
Revised: Lemma 2.

Original: For any episode k in the second phase, the policy ˆ π has the most accumulated value in Π during episodes 1 to k .
Revised: For any episode k in the second phase, the policy ˆ π has the most accumulated value in Π during episodes 1 to k .

Original: That is, ˆ π ∈ arg max π ∈ Π (cid:80) kκ =1 V πκ .
Revised: That is, ˆ π ∈ arg max π ∈ Π (cid:80) kκ =1 V πκ .

Original: Since ˆ π is the optimal policy in Π , the regret can be written as the sum of the deviation between the value of the selected policy π s and V ˆ πs
Revised: Since ˆ π is the optimal policy in Π , the regret can be written as the sum of the deviation between the value of the selected policy π s and V ˆ πs

Original: [Equation]
Revised: [Equation]

Original: According to the detection condition (equation 9) in Algorithm 2, the first term can be upper bounded by O (cid:0) √ f K C 1 k 0 (cid:1) .
Revised: According to the detection condition (equation 9) in Algorithm 2, the first term can be upper bounded by O (cid:0) √ f K C 1 k 0 (cid:1) .

Original: As for the second and last term, Freedman inequality (Lemma 12) also provides an upper bound O ( C 1 k 0 ) for them.
Revised: As for the second and last term, Freedman inequality (Lemma 12) also provides an upper bound O ( C 1 k 0 ) for them.

Original: Above all, the regret in a single epoch can be upper bounded by O (cid:0) √ f K C 1 k 0 (cid:1) .
Revised: Above all, the regret in a single epoch can be upper bounded by O (cid:0) √ f K C 1 k 0 (cid:1) .

Original: According to the choice of the minimal duration L in each epoch, the length kof the first phase in an epoch is at most half of that in the next epoch.
Revised: According to the choice of the minimal duration L in each epoch, the length kof the first phase in an epoch is at most half of that in the next epoch.

Original: Thus the final regret can be bounded as
Revised: Thus the final regret can be bounded as

Original: [Equation]
Revised: [Equation]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose the first BoBW algorithm for linear MDP that can simultaneously achieve
Revised: In this paper, we propose the first BoBW algorithm for linear MDP that can simultaneously achieve

Original: O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting.
Revised: O (poly log K ) regret in the stochastic setting and ˜ O ( √ K ) regret in the adversarial setting.

Original: Our approach relies on the new observation that the value function of a policy can be written as the sum of the inner products between the expected state-action visitation feature ϕ π,h and the unknown reward parameter θ h at different steps h ∈
Revised: Our approach relies on the new observation that the value function of a policy can be written as the sum of the inner products between the expected state-action visitation feature ϕ π,h and the unknown reward parameter θ h at different steps h ∈

Original: [ H ] .
Revised: [ H ] .

Original: And the problem can thus be regarded as an online linear optimization problem.
Revised: And the problem can thus be regarded as an online linear optimization problem.

Original: Apart from these BoBW results, we also propose a new analysis that can reach a high-probability regret guarantee for adversarial linear MDP, which is also the first such result in the literature.
Revised: Apart from these BoBW results, we also propose a new analysis that can reach a high-probability regret guarantee for adversarial linear MDP, which is also the first such result in the literature.

Original: An important future direction is to remove the assumption of unique optimal policy in the stochastic setting.
Revised: An important future direction is to remove the assumption of unique optimal policy in the stochastic setting.

Original: This assumption also appears in previous BoBW works for tabular MDP (Jin & Luo, 2020;
Revised: This assumption also appears in previous BoBW works for tabular MDP (Jin & Luo, 2020;

Original: Jin et al., 2021b) and linear bandits (Lee et al., 2021) which destroys the generality of the results but is challenging to be removed due to the hardness of the BoBW objective and the complex structure of MDP.
Revised: Jin et al., 2021b) and linear bandits (Lee et al., 2021) which destroys the generality of the results but is challenging to be removed due to the hardness of the BoBW objective and the complex structure of MDP.

Original: Extending the current results to the unknown transition setting is also prospective.
Revised: Extending the current results to the unknown transition setting is also prospective.

Original: This is hoped to be solved by some new techniques since the current approach highly depends on the known state-action visitation feature.
Revised: This is hoped to be solved by some new techniques since the current approach highly depends on the known state-action visitation feature.

Original: Deriving an FTRL-type algorithm for this objective is also an interesting future direction.
Revised: Deriving an FTRL-type algorithm for this objective is also an interesting future direction.

Original: It still remains open even in the simpler linear bandit setting without the transition between different states.
Revised: It still remains open even in the simpler linear bandit setting without the transition between different states.

Original: A A NALYSIS OF OP
Revised: A A NALYSIS OF OP

Original: The analysis of OP mainly follows the op problem for linear bandits (Lee et al., 2021) but with the consideration of the special structure of linear MDP.
Revised: The analysis of OP mainly follows the op problem for linear bandits (Lee et al., 2021) but with the consideration of the special structure of linear MDP.

Original: In this section, we provide some useful lemmas for OP in linear MDP.
Revised: In this section, we provide some useful lemmas for OP in linear MDP.

Original: Lemma 3.
Revised: Lemma 3.

Original: First, consider the constrained optimization problem:
Revised: First, consider the constrained optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where Σ h ( p ) = (cid:80) π p π ϕ π,h ϕ ⊤ π,h .
Revised: where Σ h ( p ) = (cid:80) π p π ϕ π,h ϕ ⊤ π,h .

Original: The optimal choice of p = p ∗ yields that:
Revised: The optimal choice of p = p ∗ yields that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Here we relaxed the constraint that p π be a valid distribution on Π to (cid:80)
Revised: Here we relaxed the constraint that p π be a valid distribution on Π to (cid:80)

Original: π p π ≤ 1 since there must be ˆ π ∗ where ˆ∆ ˆ π ∗ = 0 , so we can always add up the probability on ˆ π ∗ to make it a distribution.
Revised: π p π ≤ 1 since there must be ˆ π ∗ where ˆ∆ ˆ π ∗ = 0 , so we can always add up the probability on ˆ π ∗ to make it a distribution.

Original: Applying the KKT conditions and setting the derivatives with respect to each p π in the Lagrangian to zero, we got:
Revised: Applying the KKT conditions and setting the derivatives with respect to each p π in the Lagrangian to zero, we got:

Original: [Equation]
Revised: [Equation]

Original: where λ π and λ are the respective Lagrange multipliers for the constraints of p π ≥ 0 and (cid:80)
Revised: where λ π and λ are the respective Lagrange multipliers for the constraints of p π ≥ 0 and (cid:80)

Original: π p π ≤ 1 , thus we have λ ≥ 0 and (cid:80) π
Revised: π p π ≤ 1 , thus we have λ ≥ 0 and (cid:80) π

Original: λ π p π = 0 .
Revised: λ π p π = 0 .

Original: Multiplying the above equation with p ∗ π and summing over π ∈ Π , we got:
Revised: Multiplying the above equation with p ∗ π and summing over π ∈ Π , we got:

Original: [Equation]
Revised: [Equation]

Original: Notice that:
Revised: Notice that:

Original: [Equation]
Revised: [Equation]

Original: Plugging in this result
Revised: Plugging in this result

Original: [Equation]
Revised: [Equation]

Original: we got:
Revised: we got:

Original: [Equation]
Revised: [Equation]

Original: So that
Revised: So that

Original: [Equation]
Revised: [Equation]

Original: and λ ≤ 2 ξ dH , since (cid:80) π ∈ Π p ∗
Revised: and λ ≤ 2 ξ dH , since (cid:80) π ∈ Π p ∗

Original: π ˆ∆ π ≥ 0 .
Revised: π ˆ∆ π ≥ 0 .

Original: Above all,
Revised: Above all,

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Suppose that for any h ∈ { 1 , 2 , · · · , H } , (cid:8) ϕ π,h | π ∈ Π (cid:9) spans R d .
Revised: Suppose that for any h ∈ { 1 , 2 , · · · , H } , (cid:8) ϕ π,h | π ∈ Π (cid:9) spans R d .

Original: Denote p Π as a uniform distribution on Π and let κ ∈ (0 , 12 ) .
Revised: Denote p Π as a uniform distribution on Π and let κ ∈ (0 , 12 ) .

Original: For any G ⊂ Π , there exists distribution on q ∈ P G such that (cid:80)
Revised: For any G ⊂ Π , there exists distribution on q ∈ P G such that (cid:80)

Original: Hh =1 ∥ ϕ π,h ∥ 2Σ − 1 h ( q G,κ ) ≤ 2 dH , where q G,κ = κp Π + (1 − κ ) q and Σ h ( q G,κ ) =
Revised: Hh =1 ∥ ϕ π,h ∥ 2Σ − 1 h ( q G,κ ) ≤ 2 dH , where q G,κ = κp Π + (1 − κ ) q and Σ h ( q G,κ ) =

Original: (cid:80) π ∈ Π q G,κπ ϕ π,h ϕ ⊤ π,h .
Revised: (cid:80) π ∈ Π q G,κπ ϕ π,h ϕ ⊤ π,h .

Original: Proof.
Revised: Proof.

Original: Denote P G,κ = (cid:8) κp Π + (1 − κ ) q | q ∈ P G (cid:9) .
Revised: Denote P G,κ = (cid:8) κp Π + (1 − κ ) q | q ∈ P G (cid:9) .

Original: min q ∈ P G,κ max π ∈ G
Revised: min q ∈ P G,κ max π ∈ G

Original: H (cid:88) h = ∥
Revised: H (cid:88) h = ∥

Original: ϕ π,h ∥
Revised: ϕ π,h ∥

Original: 2Σ − 1 h ( q ) =
Revised: 2Σ − 1 h ( q ) =

Original: min q ∈ P G,κ max p ∈ P G
Revised: min q ∈ P G,κ max p ∈ P G

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: Tr  (cid:88)
Revised: Tr  (cid:88)

Original: π ∈ G p π ϕ π,h ϕ ⊤ π,h 
Revised: π ∈ G p π ϕ π,h ϕ ⊤ π,h 

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: π ∈ Π
Revised: π ∈ Π

Original: q π ϕ π,h ϕ ⊤ π,h   −= max p ∈ P
Revised: q π ϕ π,h ϕ ⊤ π,h   −= max p ∈ P

Original: G min q ∈ P G,κ
Revised: G min q ∈ P G,κ

Original: H (cid:88) h
Revised: H (cid:88) h

Original: =
Revised: =

Original: Tr  (cid:88)
Revised: Tr  (cid:88)

Original: π ∈ G p π ϕ π,h ϕ ⊤ π,h 
Revised: π ∈ G p π ϕ π,h ϕ ⊤ π,h 

Original:   (cid:88)
Revised:   (cid:88)

Original: π ∈ Π
Revised: π ∈ Π

Original: [Equation]
Revised: [Equation]

Original: Where the second inequality is due to Sion’s minimax theorem as equation 14 is linear in p and convex in q .
Revised: Where the second inequality is due to Sion’s minimax theorem as equation 14 is linear in p and convex in q .

Original: The last inequality is due to 1 − κ ≥ 12 .
Revised: The last inequality is due to 1 − κ ≥ 12 .

Original: Lemma 5.
Revised: Lemma 5.

Original: Let p π be the solution of OP ( k, ˆ∆) , then we have :
Revised: Let p π be the solution of OP ( k, ˆ∆) , then we have :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We transform p ∗ in Lemma 3 to a solution satisfying OP .
Revised: We transform p ∗ in Lemma 3 to a solution satisfying OP .

Original: Choosing ξ as √ kβ k in Lemma 4, and let G = { π ∈ Π : ˆ ∆ π ≤ 1 √ k } , construct the distribution q = 12 p ∗ + 12 q G,κ , where q G,κ is the distribution stated above with κ = 1 √ k in Lemma 4, we have for π ∈ G ,
Revised: Choosing ξ as √ kβ k in Lemma 4, and let G = { π ∈ Π : ˆ ∆ π ≤ 1 √ k } , construct the distribution q = 12 p ∗ + 12 q G,κ , where q G,κ is the distribution stated above with κ = 1 √ k in Lemma 4, we have for π ∈ G ,

Original: [Equation]
Revised: [Equation]

Original: and for π / ∈ G ,
Revised: and for π / ∈ G ,

Original: H (cid:88)
Revised: H (cid:88)

Original: [Equation]
Revised: [Equation]

Original: So distribution q satisfy the constraints of OP .
Revised: So distribution q satisfy the constraints of OP .

Original: For the optimal solution p of OP , we have:
Revised: For the optimal solution p of OP , we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma 6.
Revised: Lemma 6.

Original: Given { ˆ∆ π } π ∈ Π , suppose there exists unique ˆ π such that ˆ∆
Revised: Given { ˆ∆ π } π ∈ Π , suppose there exists unique ˆ π such that ˆ∆

Original: ˆ π = 0 , and ˆ∆ min = min ˆ∆ π > 

Revised: ˆ π = 0 , and ˆ∆ min = min ˆ∆ π >

Original: ˆ∆
Revised: ˆ∆

Original: π .
Revised: π .

Original: Let p π be the solution of OP ( k, ∆) , when k ≥ 16 dHβ k ˆ∆ 2min , we have (cid:80) π ∈ Π p π ˆ∆ π ≤ 24 dHβ k ˆ∆ min k .
Revised: Let p π be the solution of OP ( k, ∆) , when k ≥ 16 dHβ k ˆ∆ 2min , we have (cid:80) π ∈ Π p π ˆ∆ π ≤ 24 dHβ k ˆ∆ min k .

Original: Proof.
Revised: Proof.

Original: let G i = { π ∈ Π : 2 i − 1 ˆ∆ 2min ≤ ˆ∆ 2 π ≤ 2 i ˆ∆ 2min } and n be the largest index that G
Revised: let G i = { π ∈ Π : 2 i − 1 ˆ∆ 2min ≤ ˆ∆ 2 π ≤ 2 i ˆ∆ 2min } and n be the largest index that G

Original: i is not empty.
Revised: i is not empty.

Original: Define z i =
Revised: Define z i =

Original: dHβ k 2 i − 2 ˆ∆
Revised: dHβ k 2 i − 2 ˆ∆

Original: 2min k and κ = 1 n 2 n .
Revised: 2min k and κ = 1 n 2 n .

Original: Define the distribution ˜ p as follows:
Revised: Define the distribution ˜ p as follows:

Original: For π ̸ = ˆ π ,
Revised: For π ̸ = ˆ π ,

Original: [Equation]
Revised: [Equation]

Original: ; for ˆ π ,
Revised: ; for ˆ π ,

Original: [Equation]
Revised: [Equation]

Original: We show it’s a valid distribution over Π :
Revised: We show it’s a valid distribution over Π :

Original: [Equation]
Revised: [Equation]

Original: i ≥
Revised: i ≥

Original: [Equation]
Revised: [Equation]

Original: i ≥
Revised: i ≥

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ∈ G i (cid:88)
Revised: ∈ G i (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ̸
Revised: ̸

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: G i (cid:88)
Revised: G i (cid:88)

Original: j ̸
Revised: j ̸

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Where the last inequality is due to k ≥ 16 dHβ k ˆ∆ 2min .
Revised: Where the last inequality is due to k ≥ 16 dHβ k ˆ∆ 2min .

Original: For π ̸ = ˆ π and π ∈ G i ,
Revised: For π ̸ = ˆ π and π ∈ G i ,

Original: [Equation]
Revised: [Equation]

Original: For ˆ π ,
Revised: For ˆ π ,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Cauchy inequality.
Revised: where the last inequality is due to Cauchy inequality.

Original: So that:
Revised: So that:

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: ˜ p satisfy the constraints of OP .
Revised: ˜ p satisfy the constraints of OP .

Original: Now we bound the result of OP .
Revised: Now we bound the result of OP .

Original: By the optimality of p π :
Revised: By the optimality of p π :

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.
Revised: Lemma 7.

Original: Suppose that ˆ∆ π ∈ (cid:16) 1 √ r ∆ π , √ r ∆ π (cid:17) , then the solution p π of OP ( k, ˆ∆) for k ≥ 16 rdHβ k ∆ 2min yields that :
Revised: Suppose that ˆ∆ π ∈ (cid:16) 1 √ r ∆ π , √ r ∆ π (cid:17) , then the solution p π of OP ( k, ˆ∆) for k ≥ 16 rdHβ k ∆ 2min yields that :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the condition on ∆ π , we have t ≥ 16 rdHβ k ∆
Revised: By the condition on ∆ π , we have t ≥ 16 rdHβ k ∆

Original: [Equation]
Revised: [Equation]

Original: ≥ 

Revised: ≥

Original: dHβ k ˆ∆ 2min , and that ∆ π ∗ = ˆ∆
Revised: dHβ k ˆ∆ 2min , and that ∆ π ∗ = ˆ∆

Original: π ∗ = 0 .
Revised: π ∗ = 0 .

Original: Thus, (cid:88)
Revised: Thus, (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Where the inequality is due to Lemma 6.
Revised: Where the inequality is due to Lemma 6.

Original: B A NALYSIS IN THE S TOCHASTIC S ETTING
Revised: B A NALYSIS IN THE S TOCHASTIC S ETTING

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: First, we show the following properties, for any k in phase 1:
Revised: First, we show the following properties, for any k in phase 1:

Original: [Equation]
Revised: [Equation]

Original: Here we denote DEV
Revised: Here we denote DEV

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Claim 1’s proof: Let k = max { 900 f K C 1 ∆ 2min , L } and assume that phase 1 has not finished at episode k .
Revised: Claim 1’s proof: Let k = max { 900 f K C 1 ∆ 2min , L } and assume that phase 1 has not finished at episode k .

Original: Set
Revised: Set

Original: ˆ π =
Revised: ˆ π =

Original: π ∗ , and we show that the termination conditions hold with high probability at episode k .
Revised: π ∗ , and we show that the termination conditions hold with high probability at episode k .

Original: According to the Azuma-Hoeffding inequality, since Y s − V π s s is a martingale sequence and that
Revised: According to the Azuma-Hoeffding inequality, since Y s − V π s s is a martingale sequence and that

Original: [Equation]
Revised: [Equation]

Original: so equation 3 is satisfied.
Revised: so equation 3 is satisfied.

Original: For all π ̸ = π ∗
Revised: For all π ̸ = π ∗

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Since k ≥ 900 f K C 1 ∆ 2 π , thus k ∆ π ≥ 30 √ f K C 1 k for all π ̸ = π ∗ .
Revised: Since k ≥ 900 f K C 1 ∆ 2 π , thus k ∆ π ≥ 30 √ f K C 1 k for all π ̸ = π ∗ .

Original: So − 0 .
Revised: So − 0 .

Original: 95 k ∆ π + 2 . 1 √ f K C 1 k ≤ − 25 √ f K C 1 k .
Revised: 95 k ∆ π + 2 . 1 √ f K C 1 k ≤ − 25 √ f K C 1 k .

Original: Thus:
Revised: Thus:

Original: [Equation]
Revised: [Equation]

Original: So equation 4 is satisfied.
Revised: So equation 4 is satisfied.

Original: Claim 2’s proof: Using equation 3 and equation 4, we got:
Revised: Claim 2’s proof: Using equation 3 and equation 4, we got:

Original: [Equation]
Revised: [Equation]

Original: However, with probability at least 1 − δ , for any π ̸ = π ∗
Revised: However, with probability at least 1 − δ , for any π ̸ = π ∗

Original: [Equation]
Revised: [Equation]

Original: So we must have ˆ π = π ∗ .
Revised: So we must have ˆ π = π ∗ .

Original: Claim 3’s proof: Suppose that k 0 ≤ 64 f K C 1 ∆ 2min .
Revised: Claim 3’s proof: Suppose that k 0 ≤ 64 f K C 1 ∆ 2min .

Original: Let π be the policy with minimal estimated gap, that is: ∆ π = ∆ min .
Revised: Let π be the policy with minimal estimated gap, that is: ∆ π = ∆ min .

Original: [Equation]
Revised: [Equation]

Original: Which contradicts with equation 16.
Revised: Which contradicts with equation 16.

Original: Claim 4’s proof:
Revised: Claim 4’s proof:

Original: [Equation]
Revised: [Equation]

Original: The last inequality is due to k 0 ≥ 64 f K C 1 ∆ 2min .
Revised: The last inequality is due to k 0 ≥ 64 f K C 1 ∆ 2min .

Original: So we have:
Revised: So we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma 8.
Revised: Lemma 8.

Original: With probability at least 1 − δ , phase 2 never ends.
Revised: With probability at least 1 − δ , phase 2 never ends.

Original: Proof.
Revised: Proof.

Original: For equation 9, we decompose it as
Revised: For equation 9, we decompose it as

Original: [Equation]
Revised: [Equation]

Original: First we deal with the second term, which is a martingale difference sequence.
Revised: First we deal with the second term, which is a martingale difference sequence.

Original: It’s variance is bounded as:
Revised: It’s variance is bounded as:

Original: [Equation]
Revised: [Equation]

Original: The third term is also a martingale difference sequence
Revised: The third term is also a martingale difference sequence

Original: [Equation]
Revised: [Equation]

Original: whose variance can be bounded as:
Revised: whose variance can be bounded as:

Original: [Equation]
Revised: [Equation]

Original: Thus, using the Freedman inequality for the second and third term
Revised: Thus, using the Freedman inequality for the second and third term

Original: [Equation]
Revised: [Equation]

Original: we got:
Revised: we got:

Original: [Equation]
Revised: [Equation]

Original: For the first term
Revised: For the first term

Original: [Equation]
Revised: [Equation]

Original: we bound it’s variance against it’s expectation (cid:80)
Revised: we bound it’s variance against it’s expectation (cid:80)

Original: π ̸
Revised: π ̸

Original: =ˆ π ˜ p
Revised: =ˆ π ˜ p

Original: s ( π ) (cid:0) V
Revised: s ( π ) (cid:0) V

Original: ˆ πs − V πs
Revised: ˆ πs − V πs

Original: cid:1) as follows:
Revised: cid:1) as follows:

Original: [Equation]
Revised: [Equation]

Original: Using Freedman inequality for the martingale difference sequence above:
Revised: Using Freedman inequality for the martingale difference sequence above:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is conditioned on that π ∗
Revised: where the last inequality is conditioned on that π ∗

Original: =
Revised: =

Original: ˆ π .
Revised: ˆ π .

Original: Using Lemma 7 and equation
Revised: Using Lemma 7 and equation

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we have: k
Revised: we have: k

Original: [Equation]
Revised: [Equation]

Original: where inequality 21 is due to claim 3 that k 0 ≥ 64 f K C 1 ∆ 2min ≥
Revised: where inequality 21 is due to claim 3 that k 0 ≥ 64 f K C 1 ∆ 2min ≥

Original: [Equation]
Revised: [Equation]

Original: 2min , inequality 20 is due to ˆ∆ ∈ [0 .
Revised: 2min , inequality 20 is due to ˆ∆ ∈ [0 .

Original: 7∆ , 1 . 3∆] which is claim 4; and f K C 1 ≥ dH 2 β K log( k ) .
Revised: 7∆ , 1 . 3∆] which is claim 4; and f K C 1 ≥ dH 2 β K log( k ) .

Original: So equation 9 is never satisfied.
Revised: So equation 9 is never satisfied.

Original: According to equation 28 and equation 30, we have:
Revised: According to equation 28 and equation 30, we have:

Original: [Equation]
Revised: [Equation]

Original: Where the last inequality is due to equation 27.
Revised: Where the last inequality is due to equation 27.

Original: So:
Revised: So:

Original: [Equation]
Revised: [Equation]

Original: Thus:
Revised: Thus:

Original: [Equation]
Revised: [Equation]

Original: So equation 8 is never satisfied
Revised: So equation 8 is never satisfied

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2 .
Revised: Proof of Theorem 2 .

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we bound the deviation between the actual regret and the real regret in phase 2.
Revised: we bound the deviation between the actual regret and the real regret in phase 2.

Original: Using Freedman inequality on the martingale difference sequence ∆ π s − (cid:80) π ˜ p s ( π )∆ π :
Revised: Using Freedman inequality on the martingale difference sequence ∆ π s − (cid:80) π ˜ p s ( π )∆ π :

Original: [Equation]
Revised: [Equation]

Original: Denote M be the episode that first satisfy M ≥ 48 dHβ M ∆ 2min .
Revised: Denote M be the episode that first satisfy M ≥ 48 dHβ M ∆ 2min .

Original: For k ≥ M , we have:
Revised: For k ≥ M , we have:

Original: [Equation]
Revised: [Equation]

Original: where the inequality is due to Lemma 7 with r = 3 .
Revised: where the inequality is due to Lemma 7 with r = 3 .

Original: For k < M , according to Lemma 5 we have:
Revised: For k < M , according to Lemma 5 we have:

Original: [Equation]
Revised: [Equation]

Original: During phase
Revised: During phase

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Since we condition on that phase 2 never ends
Revised: Since we condition on that phase 2 never ends

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: C A NALYSIS IN THE A DVERSARIAL S ETTING
Revised: C A NALYSIS IN THE A DVERSARIAL S ETTING

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: First, we bound the deviation of estimation in Phase 1.
Revised: First, we bound the deviation of estimation in Phase 1.

Original: For any episode k in phase 1, we have:
Revised: For any episode k in phase 1, we have:

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:112) C 1 K − ( C 2 − 1) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k (cid:88) s
Revised: ≤ (cid:112) C 1 K − ( C 2 − 1) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k (cid:88) s

Original: = ˆ V πs − V πs (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + k (cid:88) s
Revised: = ˆ V πs − V πs (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + k (cid:88) s

Original: = (cid:16) V πs − ˆ V πs (cid:17) .
Revised: = (cid:16) V πs − ˆ V πs (cid:17) .

Original: Thus: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k (cid:88) s
Revised: Thus: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k (cid:88) s

Original: = ˆ V πs − V πs
Revised: = ˆ V πs − V πs

Original: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤
Revised: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: C 2 − (cid:112)
Revised: C 2 − (cid:112)

Original: C 1 K + k (cid:88) s =
Revised: C 1 K + k (cid:88) s =

Original: V π s s − ˆ V πs 
Revised: V π s s − ˆ V πs 

Original:  .
Revised:  .

Original: At time k 0 : (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k 0 (cid:88) s =
Revised: At time k 0 : (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k 0 (cid:88) s =

Original: ˆ V πs − V πs (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤
Revised: ˆ V πs − V πs (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: C 2 − (cid:112)
Revised: C 2 − (cid:112)

Original: C 1 K + k 0 (cid:88) s =
Revised: C 1 K + k 0 (cid:88) s =

Original: V π s s − ˆ V πs   ≤
Revised: V π s s − ˆ V πs   ≤

Original: C 2 −  2 (cid:112) C 1 K + k 0 (cid:88) s =
Revised: C 2 −  2 (cid:112) C 1 K + k 0 (cid:88) s =

Original: Y s − ˆ V πs   ≤
Revised: Y s − ˆ V πs   ≤

Original: C 2 −  2 (cid:112) C 1 K + 5 (cid:112) f K C 1 k 0 + k 0 (cid:88) s
Revised: C 2 −  2 (cid:112) C 1 K + 5 (cid:112) f K C 1 k 0 + k 0 (cid:88) s

Original: =
Revised: =

Original: ˆ V ˆ
Revised: ˆ V ˆ

Original: [Equation]
Revised: [Equation]

Original: Next, we bound the deviation of ( k − k 0 )
Revised: Next, we bound the deviation of ( k − k 0 )

Original: Rob k,π for π ̸
Revised: Rob k,π for π ̸

Original: = ˆ π .
Revised: = ˆ π .

Original: The variance of ˆ V πκ is bounded as follows:
Revised: The variance of ˆ V πκ is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Using the properties of the Catoni estimator
Revised: Using the properties of the Catoni estimator

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Where equation 26 is by our choice of α πk .
Revised: Where equation 26 is by our choice of α πk .

Original: Since equation 3 and equation 4 provides that:
Revised: Since equation 3 and equation 4 provides that:

Original: [Equation]
Revised: [Equation]

Original: we have 9 dH ≤ k 0 ˆ∆
Revised: we have 9 dH ≤ k 0 ˆ∆

Original: π β K ≤
Revised: π β K ≤

Original: [Equation]
Revised: [Equation]

Original: ˆ∆
Revised: ˆ∆

Original: 2 π
Revised: 2 π

Original: [Equation]
Revised: [Equation]

Original: Thus we have:
Revised: Thus we have:

Original: [Equation]
Revised: [Equation]

Original: Combining terms, we have:
Revised: Combining terms, we have:

Original: [Equation]
Revised: [Equation]

Original: Finally, we bound the deviation of ˆ V ˆ π .
Revised: Finally, we bound the deviation of ˆ V ˆ π .

Original: In the first k 0 episodes, since ˆ∆ ˆ π = 0 , according to equation 24, we have that:
Revised: In the first k 0 episodes, since ˆ∆ ˆ π = 0 , according to equation 24, we have that:

Original: [Equation]
Revised: [Equation]

Original: In phase 2, since E (cid:104) ˆ V ˆ πk (cid:105) =
Revised: In phase 2, since E (cid:104) ˆ V ˆ πk (cid:105) =

Original: V ˆ πk is an unbiased estimator of the true value function, according to
Revised: V ˆ πk is an unbiased estimator of the true value function, according to

Original: Freedman inequality, and that E (cid:20)(cid:16) ˆ V ˆ πk (cid:17) 2 (cid:21) =
Revised: Freedman inequality, and that E (cid:20)(cid:16) ˆ V ˆ πk (cid:17) 2 (cid:21) =

Original: E (cid:104) Y 2 k ˜ p
Revised: E (cid:104) Y 2 k ˜ p

Original: 2 k (ˆ π ) 1 { π k =
Revised: 2 k (ˆ π ) 1 { π k =

Original: ˆ π }
Revised: ˆ π }

Original: (cid:105) ≤ H 2 ˜ p k
Revised: (cid:105) ≤ H 2 ˜ p k

Original: (ˆ π ) ≤ 2 H 2 , we have:
Revised: (ˆ π ) ≤ 2 H 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: Combining the two terms, we have:
Revised: Combining the two terms, we have:

Original: [Equation]
Revised: [Equation]

Original: In sum, k
Revised: In sum, k

Original: [Equation]
Revised: [Equation]

Original: where the second to last inequality is due to equation 27, and the last one is due to equation 8.
Revised: where the second to last inequality is due to equation 27, and the last one is due to equation 8.

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Finally, we proof the regret bound in adversarial setting.
Revised: Finally, we proof the regret bound in adversarial setting.

Original: For the regret in phase
Revised: For the regret in phase

Original: [Equation]
Revised: [Equation]

Original: we can decompose it as follows.
Revised: we can decompose it as follows.

Original: [Equation]
Revised: [Equation]

Original: The first term is bounded by equation 9:
Revised: The first term is bounded by equation 9:

Original: [Equation]
Revised: [Equation]

Original: The second term is a martingale difference sequence since:
Revised: The second term is a martingale difference sequence since:

Original: [Equation]
Revised: [Equation]

Original: =0 .
Revised: =0 .

Original: It’s variance is bounded as:
Revised: It’s variance is bounded as:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to
Revised: where the last inequality is due to

Original: ˜ p s (ˆ π ) ≥ 12 .
Revised: ˜ p s (ˆ π ) ≥ 12 .

Original: The third term is also a martingale difference sequence
Revised: The third term is also a martingale difference sequence

Original: [Equation]
Revised: [Equation]

Original: whose variance is bounded as:
Revised: whose variance is bounded as:

Original: [Equation]
Revised: [Equation]

Original: Also, we have:
Revised: Also, we have:

Original: [Equation]
Revised: [Equation]

Original: Thus, using Freedman inequality on the last two terms, we got:
Revised: Thus, using Freedman inequality on the last two terms, we got:

Original: [Equation]
Revised: [Equation]

Original: Combining with the regret in phase 1, the regret in one epoch is bounded as:
Revised: Combining with the regret in phase 1, the regret in one epoch is bounded as:

Original: [Equation]
Revised: [Equation]

Original: Since the duration time k 0 of phase 1 is at least twice as the length of phase 1 in the previous epoch, we have that the sum of √ k 0 in all the epochs is bounded by an constant factor of the square root of duration time in phase 1 of the last epoch, which is bounded by √ K .
Revised: Since the duration time k 0 of phase 1 is at least twice as the length of phase 1 in the previous epoch, we have that the sum of √ k 0 in all the epochs is bounded by an constant factor of the square root of duration time in phase 1 of the last epoch, which is bounded by √ K .

Original: Summation over all the epochs, we have :
Revised: Summation over all the epochs, we have :

Original: [Equation]
Revised: [Equation]

Original: 1: Input: policy set Π , γ = min
Revised: 1: Input: policy set Π , γ = min

Original: [Equation]
Revised: [Equation]

Original: 2: Initialize: ∀ π ∈ Π , w 1 ( π ) = 1 , W 1 := | Π | .
Revised: 2: Initialize: ∀ π ∈ Π , w 1 ( π ) = 1 , W 1 := | Π | .

Original: ∀ h from 1 to H , compute the G-optimal design g h ( π ) on the set of feature visitations: { ϕ π,h , π ∈ Π } .
Revised: ∀ h from 1 to H , compute the G-optimal design g h ( π ) on the set of feature visitations: { ϕ π,h , π ∈ Π } .

Original: Denote g ( π ) =
Revised: Denote g ( π ) =

Original: 1 H (cid:80)
Revised: 1 H (cid:80)

Original: Hh =1 g h ( π ) 3: for each episode k
Revised: Hh =1 g h ( π ) 3: for each episode k

Original: = 1 to K do 4:
Revised: = 1 to K do 4:

Original: [Equation]
Revised: [Equation]

Original: Select π k ∈ Π according to the probability p k
Revised: Select π k ∈ Π according to the probability p k

Original: ( π ) and collect rewards y k,h 6:
Revised: ( π ) and collect rewards y k,h 6:

Original: Calculate reward estimators: ˆ θ k,h
Revised: Calculate reward estimators: ˆ θ k,h

Original: =
Revised: =

Original: Σ − 1 k,h ϕ π k ,h y k,h , and ˆ r πk,h = ϕ
Revised: Σ − 1 k,h ϕ π k ,h y k,h , and ˆ r πk,h = ϕ

Original: ⊤ π,h ˆ θ k,h , ˆ V πk
Revised: ⊤ π,h ˆ θ k,h , ˆ V πk

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Hh =1 ˆ r πk,h , where Σ k,h = (cid:80)
Revised: Hh =1 ˆ r πk,h , where Σ k,h = (cid:80)

Original: π p k
Revised: π p k

Original: ( π ) ϕ π,h ϕ ⊤ π,h 7:
Revised: ( π ) ϕ π,h ϕ ⊤ π,h 7:

Original: Compute the optimistic estimate of the value function:
Revised: Compute the optimistic estimate of the value function:

Original: [Equation]
Revised: [Equation]

Original:  8:
Revised:  8:

Original: And we transform it into loss functions: l k,h ( s h , a h ) =
Revised: And we transform it into loss functions: l k,h ( s h , a h ) =

Original: 1 − r k,h ( s h , a h ) , l πk,h = 1 − r πk,h
Revised: 1 − r k,h ( s h , a h ) , l πk,h = 1 − r πk,h

Original: L πk,h =
Revised: L πk,h =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: = l πk,h
Revised: = l πk,h

Original: = H − V πk,h ˆ L
Revised: = H − V πk,h ˆ L

Original: [Equation]
Revised: [Equation]

Original: 9: update using the loss estimators:
Revised: 9: update using the loss estimators:

Original: [Equation]
Revised: [Equation]

Original: 10: end for
Revised: 10: end for

Original: D H IGH P ROBABILITY G UARANTEE FOR A DVERSARIAL L INEAR MDP
Revised: D H IGH P ROBABILITY G UARANTEE FOR A DVERSARIAL L INEAR MDP

Original: First, we bound the deviation between the estimated value and the true value of policy π .
Revised: First, we bound the deviation between the estimated value and the true value of policy π .

Original: Lemma 9.
Revised: Lemma 9.

Original: Denote DEV K,π = (cid:12)(cid:12)(cid:12)(cid:80)
Revised: Denote DEV K,π = (cid:12)(cid:12)(cid:12)(cid:80)

Original: Kk =1 ˆ V πk − V πk (cid:12)(cid:12)(cid:12) , then we have:
Revised: Kk =1 ˆ V πk − V πk (cid:12)(cid:12)(cid:12) , then we have:

Original: DEV K,π = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: DEV K,π = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = ˆ V πk − V πk (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
Revised: = ˆ V πk − V πk (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = ˆ L πk − L πk (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤
Revised: = ˆ L πk − L πk (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: C
Revised: C

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: H (cid:88) h
Revised: H (cid:88) h

Original: = (cid:13)(cid:13) ϕ π,h (cid:13)(cid:13) 2 Σ − 1 k,h (cid:115)
Revised: = (cid:13)(cid:13) ϕ π,h (cid:13)(cid:13) 2 Σ − 1 k,h (cid:115)

Original: H log (cid:0) 1 δ (cid:1) dK + C(cid:115) dKH log (cid:18)δ (cid:19) +(cid:32)
Revised: H log (cid:0) 1 δ (cid:1) dK + C(cid:115) dKH log (cid:18)δ (cid:19) +(cid:32)

Original: dHγ + H (cid:33) log (cid:18)δ (cid:19) .
Revised: dHγ + H (cid:33) log (cid:18)δ (cid:19) .

Original: Proof.
Revised: Proof.

Original: First, we show that ˆ V πk is an unbiased estimate of V πk :
Revised: First, we show that ˆ V πk is an unbiased estimate of V πk :

Original: [Equation]
Revised: [Equation]

Original: using the tower rule of expectation, we have:
Revised: using the tower rule of expectation, we have:

Original: [Equation]
Revised: [Equation]

Original: so
Revised: so

Original: [Equation]
Revised: [Equation]

Original: Denote σ
Revised: Denote σ

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , then:
Revised: (cid:17) , then:

Original: [Equation]
Revised: [Equation]

Original: Also, due to the properties of G-optimal design, we have:
Revised: Also, due to the properties of G-optimal design, we have:

Original: [Equation]
Revised: [Equation]

Original: and Σ k,h ⪰ γH (cid:80) π g h ( π ) ϕ π,h ϕ ⊤ π,h .
Revised: and Σ k,h ⪰ γH (cid:80) π g h ( π ) ϕ π,h ϕ ⊤ π,h .

Original: Thus we have (cid:13)(cid:13) ϕ π,h (cid:13)(cid:13) 2 Σ − 1 k,h ≤ dHγ , ∀ π ∈ Π , so
Revised: Thus we have (cid:13)(cid:13) ϕ π,h (cid:13)(cid:13) 2 Σ − 1 k,h ≤ dHγ , ∀ π ∈ Π , so

Original: ˆ V πk ≤ dH 2 γ .
Revised: ˆ V πk ≤ dH 2 γ .

Original: Using
Revised: Using

Original: Freedman inequality, the sum of the martingale difference sequence
Revised: Freedman inequality, the sum of the martingale difference sequence

Original: ˆ V πk − V πk is bounded as:
Revised: ˆ V πk − V πk is bounded as:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the geometric mean-arithmetic mean inequality.
Revised: where the last inequality is due to the geometric mean-arithmetic mean inequality.

Original: Lemma 10.
Revised: Lemma 10.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: K
Revised: K

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using lemma 6 in Bartlett et al.
Revised: Using lemma 6 in Bartlett et al.

Original: (2008), we have:
Revised: (2008), we have:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:12)(cid:12)(cid:12) ˆ θ ⊤ k,h ϕ π,h (cid:12)(cid:12)(cid:12) ≤ dHγ .
Revised: Since (cid:12)(cid:12)(cid:12) ˆ θ ⊤ k,h ϕ π,h (cid:12)(cid:12)(cid:12) ≤ dHγ .

Original: Plugging it into equation 33, we finish the proof.
Revised: Plugging it into equation 33, we finish the proof.

Original: Lemma 11.
Revised: Lemma 11.

Original: With probability at least 1 − δ , we have :
Revised: With probability at least 1 − δ , we have :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since ˆ L πk =
Revised: Since ˆ L πk =

Original: H − ˆ V πk , (cid:16) ˆ L πk (cid:17) 2 ≤ (cid:16)
Revised: H − ˆ V πk , (cid:16) ˆ L πk (cid:17) 2 ≤ (cid:16)

Original: ˆ V πk (cid:17) 

Revised: ˆ V πk (cid:17)

Original: + H 2 , we have:
Revised: + H 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: Using Cauchy-Schwarz inequality, we have:
Revised: Using Cauchy-Schwarz inequality, we have:

Original: [Equation]
Revised: [Equation]

Original: So,
Revised: So,

Original: [Equation]
Revised: [Equation]

Original: Notice that using the definition of Σ k,h and properties of the G-optimal design,
Revised: Notice that using the definition of Σ k,h and properties of the G-optimal design,

Original: [Equation]
Revised: [Equation]

Original: Applying the Hoeffding bound
Revised: Applying the Hoeffding bound

Original: [Equation]
Revised: [Equation]

Original: we got:
Revised: we got:

Original: [Equation]
Revised: [Equation]

Original: Thus:
Revised: Thus:

Original: [Equation]
Revised: [Equation]

Original: And:
Revised: And:

Original: [Equation]
Revised: [Equation]

Original: So we conclude that:
Revised: So we conclude that:

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1 .
Revised: Proof of Theorem 1 .

Original: Now we analyze the potential function.
Revised: Now we analyze the potential function.

Original: Using classical techniques
Revised: Using classical techniques

Original: [Equation]
Revised: [Equation]

Original: we got that counter part of equation (2) in Bartlett et al.
Revised: we got that counter part of equation (2) in Bartlett et al.

Original: [Equation]
Revised: [Equation]

Original: where inequality 34 is due to the constraint that (cid:12)(cid:12)(cid:12)
Revised: where inequality 34 is due to the constraint that (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 10, we have:
Revised: Using Lemma 10, we have:

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 9 and choosing C 2 =
Revised: Using Lemma 9 and choosing C 2 =

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Thus equation 35 becomes:
Revised: Thus equation 35 becomes:

Original: [Equation]
Revised: [Equation]

Original: k 
Revised: k 

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: δ (cid:19) + 8 dH
Revised: δ (cid:19) + 8 dH

Original: [Equation]
Revised: [Equation]

Original: log (cid:0) 1 δ (cid:1) γ   + 2 η 
Revised: log (cid:0) 1 δ (cid:1) γ   + 2 η 

Original: [Equation]
Revised: [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:17)
Revised: 1 (cid:17)

Original: [Equation]
Revised: [Equation]

Original: log (cid:18)δ (cid:19) +
Revised: log (cid:18)δ (cid:19) +

Original: [Equation]
Revised: [Equation]

Original: (cid:32)
Revised: (cid:32)

Original: H + dHγ (cid:33) log (cid:18)δ (cid:19) + 2 H (cid:115) dKH log (cid:18)δ (cid:19)  + 2 ηγKH + η 
Revised: H + dHγ (cid:33) log (cid:18)δ (cid:19) + 2 H (cid:115) dKH log (cid:18)δ (cid:19)  + 2 ηγKH + η 

Original: [Equation]
Revised: [Equation]

Original: (cid:115) dKH log (cid:18)δ (cid:19)
Revised: (cid:115) dKH log (cid:18)δ (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:32) dH
Revised: (cid:32) dH

Original: [Equation]
Revised: [Equation]

Original: γ + H (cid:33) log (cid:18)δ (cid:19) .
Revised: γ + H (cid:33) log (cid:18)δ (cid:19) .

Original: And we also have for ∀ π ∈ Π ,
Revised: And we also have for ∀ π ∈ Π ,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Lemma
Revised: where the last inequality is due to Lemma

Original: [Equation]
Revised: [Equation]

Original: Plugging equation 36 and equation 37 together
Revised: Plugging equation 36 and equation 37 together

Original: [Equation]
Revised: [Equation]

Original: we have that for ∀ π ∈ Π :
Revised: we have that for ∀ π ∈ Π :

Original: K
Revised: K

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When K ≥ L 0 = 4 dH log (cid:16) | Π | δ (cid:17) and γ ≤ 12 ,
Revised: When K ≥ L 0 = 4 dH log (cid:16) | Π | δ (cid:17) and γ ≤ 12 ,

Original: [Equation]
Revised: [Equation]

Original: So the choice of η = γ 4 dH 2 ensures (cid:12)(cid:12)(cid:12) η ˜ L πk (cid:12)(cid:12)(cid:12) ≤ 1 .
Revised: So the choice of η = γ 4 dH 2 ensures (cid:12)(cid:12)(cid:12) η ˜ L πk (cid:12)(cid:12)(cid:12) ≤ 1 .

Original: Plugging in our choice of η , equation 38 then becomes:
Revised: Plugging in our choice of η , equation 38 then becomes:

Original: [Equation]
Revised: [Equation]

Original: Choosing C 2 = 12 , we have:
Revised: Choosing C 2 = 12 , we have:

Original: [Equation]
Revised: [Equation]

Original: And by our choice of γ = min
Revised: And by our choice of γ = min

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: For K ≤ L 0 = 4 dH log (cid:16) | Π | δ (cid:17) , we have:
Revised: For K ≤ L 0 = 4 dH log (cid:16) | Π | δ (cid:17) , we have:

Original: [Equation]
Revised: [Equation]

Original: Combining equation 41 and equation 40, we prove the regret bound.
Revised: Combining equation 41 and equation 40, we prove the regret bound.

Original: Moreover, when K ≥ L 0 , choosing C 2 ≥ 20 and recalling the definition of DEV k,π in Lemma 9,
Revised: Moreover, when K ≥ L 0 , choosing C 2 ≥ 20 and recalling the definition of DEV k,π in Lemma 9,

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: Applying a union bound for all the possible k 0 ∈ { 1 , 2 , · · · K } , we conclude with at least probability 1 − δ , we have:
Revised: Applying a union bound for all the possible k 0 ∈ { 1 , 2 , · · · K } , we conclude with at least probability 1 − δ , we have:

Original: [Equation]
Revised: [Equation]

Original: With the constant C 1 =
Revised: With the constant C 1 =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: , proving Theorem 1.
Revised: , proving Theorem 1.

Original: E C ONCENTRATION I NEQUALITIES
Revised: E C ONCENTRATION I NEQUALITIES

Original: Lemma 12.
Revised: Lemma 12.

Original: [Freedman inequality(Freedman, 1975)] Let F 0 ⊂
Revised: [Freedman inequality(Freedman, 1975)] Let F 0 ⊂

Original: F 1 ⊂ · · · ⊂ F T be a filtration and let X 1 , X 2 , · · · X T be random variables such that X t is F t measurable, E (cid:2)
Revised: F 1 ⊂ · · · ⊂ F T be a filtration and let X 1 , X 2 , · · · X T be random variables such that X t is F t measurable, E (cid:2)

Original: X t |F t − 1 (cid:3) = 0 , | X t | ≤ b almost surely, and (cid:80) Tt =1 E (cid:2)
Revised: X t |F t − 1 (cid:3) = 0 , | X t | ≤ b almost surely, and (cid:80) Tt =1 E (cid:2)

Original: X 2 t |F t − 1 (cid:3) ≤ V for some fixed V > 0 and b > 0 .
Revised: X 2 t |F t − 1 (cid:3) ≤ V for some fixed V > 0 and b > 0 .

Original: Then, for any δ ∈ (0 , 1) , we have with probability at least 1 − δ ,
Revised: Then, for any δ ∈ (0 , 1) , we have with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 13 (Concentration inequality for Catoni estimators (Wei & Luo, 2018; Lee et al., 2021)) .
Revised: Lemma 13 (Concentration inequality for Catoni estimators (Wei & Luo, 2018; Lee et al., 2021)) .

Original: Let F 0 ⊂ F 1 ⊂ · · · ⊂ F n be a filtration and let X 1 , X 2 , · · · X n be random variables such that
Revised: Let F 0 ⊂ F 1 ⊂ · · · ⊂ F n be a filtration and let X 1 , X 2 , · · · X n be random variables such that

Original: X i is F i measurable, E (cid:2)
Revised: X i is F i measurable, E (cid:2)

Original: X i |F i − 1 (cid:3)
Revised: X i |F i − 1 (cid:3)

Original: =
Revised: =

Original: µ i for some fixed µ i , and (cid:80) ni
Revised: µ i for some fixed µ i , and (cid:80) ni

Original: =1 E (cid:104)
Revised: =1 E (cid:104)

Original: ( X i − µ i ) 2 |F i − 1 (cid:105) ≤ V for some fixed V .
Revised: ( X i − µ i ) 2 |F i − 1 (cid:105) ≤ V for some fixed V .

Original: Denote µ = 1 n (cid:80) ni
Revised: Denote µ = 1 n (cid:80) ni

Original: =1 µ i and let ˆ µ n,α be the Catoni’s robust mean estimator of X 1 , X 2 , · · · X n with a fixed parameter α , that is, ˆ µ n,α is the unique root of the function: f ( z )
Revised: =1 µ i and let ˆ µ n,α be the Catoni’s robust mean estimator of X 1 , X 2 , · · · X n with a fixed parameter α , that is, ˆ µ n,α is the unique root of the function: f ( z )

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 Φ (cid:0) α
Revised: =1 Φ (cid:0) α

Original: ( X i − z ) (cid:1) , where Φ( y )
Revised: ( X i − z ) (cid:1) , where Φ( y )

Original: = log (cid:0) 1 + y + y 2 / 2 (cid:1)
Revised: = log (cid:0) 1 + y + y 2 / 2 (cid:1)

Original: if y ≥ 0 and Φ( y ) = − log (cid:0) 1 − y + y 2 / 2 (cid:1) otherwise.
Revised: if y ≥ 0 and Φ( y ) = − log (cid:0) 1 − y + y 2 / 2 (cid:1) otherwise.

Original: Then for any δ ∈ (0 , 1) , as long as n is large enough that n ≥ α 2 (cid:16) V + (cid:80) ni
Revised: Then for any δ ∈ (0 , 1) , as long as n is large enough that n ≥ α 2 (cid:16) V + (cid:80) ni

Original: =1 ( µ i − µ ) 2 (cid:17)
Revised: =1 ( µ i − µ ) 2 (cid:17)

Original: + 2 log (cid:0) 1 /δ (cid:1) , we have with probability at least 1 − 2 δ ,
Revised: + 2 log (cid:0) 1 /δ (cid:1) , we have with probability at least 1 − 2 δ ,

Original: [Equation]
Revised: [Equation]

Original: Choosing α optimally, we have:
Revised: Choosing α optimally, we have:

Original: [Equation]
Revised: [Equation]

Original: In particular, if µ 1 = µ 2 = · · · = µ n , we have:
Revised: In particular, if µ 1 = µ 2 = · · · = µ n , we have:

Original: [Equation]
Revised: [Equation]