Original: Exploration Hurts in Bandits with Partially Observed Stochastic Contexts
Revised: A Regret Bound for Greedy Partially Observed Stochastic Contextual Bandits

Original: Abstract
Revised: Abstract

Original: Contextual bandits are widely-used models in reinforcement learning for incorporating both common and idiosyncratic factors in reward functions.
Revised: Contextual bandits are widely-used models in reinforcement learning for incorporating both generic and idiosyncratic factors in reward functions.

Original: The existing approaches rely on full observation of thestochastic context vectors, while the problem of learning optimal arms from partially observed contexts remains immature.
Revised: The existing approaches rely on full observation of the context vectors, while the problem of learning optimal arms from partially observed contexts remains immature.

Original: Wetheoretically show that in the latter setting, decisions can be made more guarded to minimize the risk of pulling sub-optimal arms.
Revised: We show that in the latter setting, decisions can be made more guarded to minimize the risk of pulling suboptimal arms.

Original: More precisely, efﬁciency is established for Greedy policies that treat the estimates of the unknown parameter and of the unobserved contexts as their true values.
Revised: More precisely, efﬁciency is established for Greedy policies that treat the estimates of the unknown parameter and of the unobserved contexts as their true values.

Original: That includes non-asymptotic worst-case regret bounds that grow (poly-)logarithmically with the time horizon andfailure probability, and linearly with the number of arms.
Revised: That includes nonasymptotic high probability regret bounds that grow logarithmically with the time horizon and linearly with the number of arms.

Original: Numerical results that showcase the efﬁcacy of avoiding exploration are provided.
Revised: Numerical results that showcase the efﬁcacy of avoiding exploration are provided as well.

Original: 1. Introduction
Revised: 1. Introduction

Original: Contextual bandits are ubiquitous models for sequential decision making in environments with ﬁnite action spaces.
Revised: Contextual bandits are ubiquitous models for sequential decision-making in environments with ﬁnite action spaces.

Original: The range of applications is extensive and includes different problems for which time-varying and action-dependent information are important, such as personalized recommendation of news articles, healthcare interventions, advertisements, and clinical trials (Li et al., 2010;
Revised: The range of applications is extensive and includes different problems for which time-varying and action-dependent information are important, such as personalized recommendation of news articles, healthcare interventions, advertisements, and clinical trials ( Li et al., 2010;

Original: Bouneffouf et al., 2012; Tewari & Murphy, 2017; Nahum-Shani et al., 2018;
Revised: Bouneffouf et al., 2012; Tewari & Murphy, 2017; Nahum-Shani et al., 2018;

Original: Durand et al., 2018; Varatharajah et al., 2018; Ren & Zhou, 2020).
Revised: Durand et al., 2018; Varatharajah et al., 2018; Ren & Zhou, 2020).

Original: In many applications, consequential variables for decision making are not perfectly observed.
Revised: In many applications consequential variables for decision-making are not perfectly observed.

Original: Technically speaking, in the bandit problem, the context vectors are often observed in a partial, transformed, and/or noisy  manner (Bensoussan,  et al., 2017;
Revised: Technically speaking, in the bandit problem, the context vectors are often observed in a partial, transformed, or noisy Proceedings manner (Bensoussan, 2004; Bouneffouf et al., 2017;

Original: Tennenholtz et al., 2021).
Revised: Tennenholtz et al., 2021).

Original: Furthermore, in general, sequential decision making algorithms under imperfect observations provide a richer class of models compared to those of perfect observations.
Revised: Furthermore, since perfect observation can be considered as a special case of imperfectly observed contexts, sequential decision-making algorithms for the latter family of problems provide a richer class of settings compared to the former.

Original: Accordingly, they are commonly used in different problems, including space-state models, robot control, and ﬁltering (Nagrath, 2006;Lin et al., 2012;
Revised: Accordingly, partial observation models are commonly used in different problems, including space-state models, robot control, and ﬁltering ( Nagrath, 2006;

Original:  Kang et al., 2012).
Revised: Lin et al., 2012; Kang et al., 2012).

Original: We study contextual bandits with imperfectly observed context vectors.
Revised: We study contextual bandits with imperfectly observed context vectors.

Original: The probabilistic structure of the problem under study as time proceeds, is as follows.
Revised: The probabilistic structure of the problem under study, as time proceeds, is as follows.

Original: At every time step, there are N available actions (also referred to as ‘arms’), and the unobserved context of arm i at time t , denoted by x i ( t ) 
Revised: At every time step, there are N available arms, each of which has the unobserved context that is denoted by x i ( t ) ∈ R d x for arm i at time t .

Original: ∈ R d x , is generated  according to a multivariate normal distribution N (0 d x , Σ x ) .
Revised: The context vectors are generated independent of the previous contexts and independent of the other arms, according to a multivariate normal distribution N (0 d x , Σ x ) .

Original: Moreover, the corresponding observation (i.e., output) is y i ( t ) ∈
Revised: Moreover, the corresponding observation of x i ( t ) is y i ( t ) ∈

Original: R d y , while the stochastic reward r i ( t ) of arm i is determined by the context vector and the unknown parameter µ ∗ . Formally, we have [Equation] [Equation]
Revised: R d y , while the stochastic reward r i ( t ) of arm i is determined by the context vector and the unknown parameter µ ∗ :

Original: [Equation]
Revised: [Equation]

Original: where ζ i ( t ) and ψ i ( t ) are the noises of observation and reward, which are identically distributed and independent following the distributions N (0 d y , Σ y ) and N (0 , γ 2 r ) , respectively.
Revised: Above, ζ i ( t ) and ψ i ( t ) are the noises of observation and reward, which are identically distributed and independent following the distributions N (0 d y , Σ y ) and N (0 , γ 2 r ) , respectively.

Original: Further, the  d y × d x sensing matrix
Revised: Further, the known d y × d x sensing matrix

Original: A captures the relationship between x i ( t ) and the noiseless portion of y i ( t ) .
Revised: A captures the relationship between x i ( t ) and the noiseless portion of y i ( t ) .

Original: The above structure holds for all arm i and at all time t .
Revised: The above structure holds for all arms and at all time t .

Original: For this model, as compared to the classic contextual bandits, there is uncertainty in contexts as well as in the environment associated with the parameter µ ∗ .
Revised: For this model, as compared to the classic contextual bandits, there is uncertainty in contexts as well as in the environment associated with the parameter µ ∗ .

Original: At each time, the goal is to learn to choose the optimal arm a ∗ ( t ) maximizing the reward, by utilizing the available information by time t .
Revised: At each time, the goal is to learn to choose the optimal arm a ∗ ( t ) (which can change at every time step), by utilizing the available information by time t .

Original: That is, the agent chooses an arm based on the previously collected data from the model in (1); { a ( τ ) } 1 ≤ τ ≤ t − 1 , { y a ( τ ) ( τ ) } 1 ≤ τ ≤ t − 1 , { r a ( τ ) } 1 ≤ τ ≤ t − 1 , as well as theoutput observations at the time; { y i ( t ) } 1 ≤ i ≤ N .
Revised: That is, the agent chooses an arm based on the previously collected data from the model in (1); { a ( τ ) } 1 ≤ τ ≤ t − 1 , { y a ( τ ) ( τ ) } 1 ≤ τ ≤ t − 1 , { r a ( τ ) } 1 ≤ τ ≤ t − 1 , as well as the observations at the time; { y i ( t ) } 1 ≤ i ≤ N .

Original: Then, once the action is taken, the resulting reward  will be provided to the agent according to the equation in (2).
Revised: Then, once the action is taken, the resulting reward of the chosen arm will be provided to the agent according to the equation in (2), while rewards of the other arms are not observed.

Original: Clearly, to choose high-reward arms, the agent needs accurate estimates of the unknown parameter µ ∗ , as well as those of the contexts x i ( t ) , for i = 1 , · · · , N .
Revised: Clearly, to choose high-reward arms, the agent needs accurate estimates of the unknown parameter µ ∗ , as well as those of the contexts x i ( t ) , for i = 1 , · · · , N .

Original: However, because x i ( t ) is not observed,the estimation of µ ∗ is available only through the output y i ( t ) .
Revised: However, because x i ( t ) is not observed, estimation of µ ∗ is feasible only through the observation y i ( t ) .

Original: Thereby, design of efﬁcient reinforcement learning policies with guaranteed performance is challenging.
Revised: Thereby, design of efﬁcient reinforcement learning policies with guaranteed performance is challenging.

Original: Learning strategies for contextual bandits are investigated in the literature, assuming that the context vectors are fully observed.
Revised: Learning strategies for contextual bandits are investigated in the literature, assuming that the context vectors are fully observed.

Original: Early papers focus on reinforcement learning policies that utilize Upper-Conﬁdent-Bounds (UCB) for addressing the exploitation-exploration trade-off (Auer, 2002;
Revised: Early papers focus on reinforcement learning policies that utilize Upper-Conﬁdent-Bounds (UCB) for addressing the exploitation-exploration trade-off (Auer, 2002;

Original: Abbasi-Yadkori et al., 2011; Chu et al., 2011).
Revised: Abbasi-Yadkori et al., 2011; Chu et al., 2011).

Original: Another popular and efﬁcient family of policies use randomized exploration, usually in the (Bayesian) form of Thompson sampling (Chapelle & Li, 2011; Agrawal & Goyal, 2013;
Revised: Another popular and efﬁcient family of policies use randomized exploration, usually in the (Bayesian) form of Thompson sampling (Chapelle & Li, 2011; Agrawal & Goyal, 2013;

Original: Faradonbeh et al., 2020; Modi & Tewari, 2020).
Revised: Faradonbeh et al., 2020; Modi & Tewari, 2020).

Original: However, forcontextual bandits that contextsare generated under certain conditions, it isrecently shown that exploration-free  policies have efﬁcient performances (Bastani et al., 2021).
Revised: Recently, for contexts generated under certain conditions, it is shown that exploration-free greedy policies have efﬁcient performances (Bastani et al., 2021).

Original: Currently,  theoretical results for imperfect context observationsare incomplete, although the model is commonly used in  applications.
Revised: Currently, study of efﬁcient algorithms with theoretical performance guarantees for imperfect context observations is rather incomplete. Notably, imperfectness of observation frequently appears in different areas of applications.

Original: The causesof imperfect observations are various, including privacy regulations (Sbeity & Younes, 2015), measurement errors (Lin et al., 2012; Kang et al., 2012), and missingness (Azimi et al., 2019).
Revised: The causes are various, including privacy regulations (Sbeity & Younes, 2015), measurement errors (Lin et al., 2012; Kang et al., 2012), and missingness (Azimi et al., 2019).

Original: For contextual bandits with noisy observations of the same dimension as that of contexts, asymptotic analyses  are available for UCB-type algorithms (Yun et al., 2017), Thompson sampling (Park & Faradonbeh, 2021),  in presence of additional information (Tennenholtz et al., 2021), and inthe adversarial setting (Lattimore & Szepesv´ari, 2019; Lattimore, 2022).
Revised: The existing analyses study some special cases, such as those with invertible sensing matrices. Asymptotic results are shown for UCB-type algorithms (Yun et al., 2017) and Thompson sampling (Park & Faradonbeh, 2021), as well as in presence of additional information (Tennenholtz et al., 2021), and in adversarial settings with partial monitoring (Lattimore & Szepesv´ari, 2019; Lattimore, 2022).

Original: Moreover, numerical analyses show that  in partially observed contextual bandits, Greedy algorithms outperform Thompson sampling (Park & Faradonbeh, 2022).
Revised: For Greedy algorithms, numerical analyses indicate that they outperform Thompson sampling in partially observed contextual bandits with stochastic contexts (Park & Faradonbeh, 2022).

Original:  Greedy policies are of special interests in settings that exploration is not (ethically) permitted, such as precision medicine (Bastani et al., 2021).
Revised: Importantly, Greedy policies are of special interests in settings that exploration is not (ethically) permitted, such as precision medicine (Bastani et al., 2021).

Original: However, comprehensive theoretical performance guarantees are unavailable.
Revised: However, comprehensive theoretical performance guarantees are currently unavailable for Greedy policies.

Original: We perform the ﬁnite-time worst-case analysis of Greedy reinforcement learning policies for imperfectly observed contextual bandits.
Revised: We perform the ﬁnite-time worst-case analysis of Greedy reinforcement learning policies for imperfectly observed contextual bandits.

Original: We provide high probability regret bounds that consists of poly-logarithmic factors of the time horizon and of the failure probability.
Revised: We provide high probability regret bounds that consists of poly-logarithmic factors of the time horizon and of the failure probability.

Original: Furthermore, the effects of other quantities such as the number of arms, dimension, and  noise processes, are fully characterized.
Revised: Furthermore, the effects of other quantities such as the number of arms, dimension, and properties of the noise processes, are fully characterized.

Original: Illustrative numerical experiments showcasing the efﬁciency of Greedy policies are also provided.
Revised: Illustrative numerical experiments showcasing the efﬁciency of Greedy policies are also provided.

Original: Different technical difﬁculties arise in the high probability analysis of reinforcement learning policies in partially observed contextual bandits.
Revised: Different technical difﬁculties arise in the high probability analysis of reinforcement learning policies in partially observed contextual bandits.

Original: First, one needs to study the eigenvalues of the empirical covariance matrices, since the estimation accuracy heavily depends on them.
Revised: First, one needs to study the eigenvalues of the empirical covariance matrices, since the estimation accuracy heavily depends on them.

Original: Furthermore, it is required to consider the number of times the algorithm selects sub-optimal arms.
Revised: Furthermore, it is required to consider the number of times the algorithm selects sub-optimal arms.

Original: Note that both quantities are stochastic and so worst-case (i.e., high probability) results are needed for a statistically dependent sequence of random objects.
Revised: Note that both quantities are stochastic and so worst-case (i.e., high probability) results are needed for a statistically dependent sequence of random objects.

Original: To obtain the presented theoretical results, we employ advanced technical tools from martingale theory and random matrices.
Revised: To obtain the presented theoretical results, we employ advanced technical tools from martingale theory and random matrices.

Original: Indeed, by utilizing concentration inequalities for matrices with martingale difference structures, we carefully characterise the effects of order statistics and tail-properties of the estimation errors.
Revised: Indeed, by utilizing concentration inequalities for matrices with martingale difference structures, we carefully characterise the effects of order statistics and tail-properties of the estimation errors.

Original: A highlight of this paper is as follows.
Revised: A highlight of this paper is as follows.

Original: In Section 2, we formulate the problem and discuss the preliminaries.
Revised: In Section 2, we formulate the problem and discuss the preliminary materials.

Original: Next, a Greedy policy for contextual bandits with imperfect context observations is presented in Section 3.
Revised: Next, a Greedy policy for contextual bandits with imperfect context observations is presented in Section 3.

Original: In Section 4, we provide theoretical performance guarantees, followed by numerical experiments in Section 5.
Revised: In Section 4, we provide theoretical performance guarantees, followed by numerical experiments in Section 5.

Original: Notation A ⊤ is the transpose of A and the ℓ 2 norm is
Revised: Notation A ⊤ is the transpose of A and the ℓ 2 norm is

Original: [Equation]
Revised: [Equation]

Original: note the minimum (maximum) eigenvalues of A , respectively.
Revised: di =1 | v i |(cid:17) 1 / 2 . Moreover, λ min ( A ) ( λ max ( A ) ) denote the minimum (maximum) eigenvalues of A , respectively.

Original: Finally, O ( · ) is the order of magnitude, { X i } i ∈ E
Revised: In addition, O ( · ) is the order of magnitude such that f ( n ) =

Revised: O ( g ( n )) denotes lim sup n →∞ | f ( n ) | /g ( n ) < ∞ for a real valued function f and a strictly positive valued function g .

Revised: Finally, { X i } i ∈ E =

Original: = { X i : i ∈ E } , and I ( · ) is the indicator function.
Revised: { X i : i ∈ E } , and I ( · ) is the indicator function.

Original: 2. Problem Formulation
Revised: 2. Problem Formulation

Original: First, we formally discuss the problem of contextual bandits with imperfect context observations.
Revised: First, we formally discuss the problem of contextual bandits with imperfect context observations.

Original: A bandit machine  has N arms, each of which has its own unobserved context x i ( t ) , for i ∈ { 1 , · · · , N } .
Revised: The bandit machine under consideration has N arms, each of which has its own unobserved context x i ( t ) , for i ∈ { 1 , · · · , N } .

Original: Equation (1) presents the observation model, where the observations { y i ( t ) } 1 ≤ i ≤ N are linearly transformed functions of the contexts, perturbed by additive noise vectors { ζ i ( t ) } 1 ≤ i ≤ N .
Revised: Equation (1) presents the observation model, where the observations { y i ( t ) } 1 ≤ i ≤ N are linearly transformed functions of the contexts, perturbed by additive noise vectors { ζ i ( t ) } 1 ≤ i ≤ N .

Original: Equation (2) describes the process of reward generation for different arms, depicting that if the agent selects arm i , then the resulting reward is an unknown linear function of the unobserved context vector, subject to some additional randomness due to the reward noise ψ i ( t ) .
Revised: Equation (2) describes the process of reward generation for different arms, depicting that if the agent selects arm i , then the resulting reward is an unknown linear function of the unobserved context vector, subject to some additional randomness due to the reward noise ψ i ( t ) .

Original: The agent aims to maximize the cumulative reward over time, by utilizing the sequence of observations.
Revised: The agent aims to maximize the cumulative reward over time, by utilizing the sequence of observations.

Original: To gain the maximum possible reward, the agent needs to learn the relationship between the rewards r i ( t ) and the observations y i ( t ) .
Revised: To gain the maximum possible reward, the agent needs to learn the relationship between the reward r i ( t ) and the observation y i ( t ) .

Original: For that purpose, we proceed by considering the conditional distribution of the reward r i ( t ) given the observation y i ( t ) , i.e., P ( r i ( t )
Revised: For that purpose, we proceed by considering the conditional distribution of the reward r i ( t ) given the observation y i ( t ) ; i.e., P ( r i ( t )

Original: |
Revised: |

Original: y
Revised: y

Original: i ( t )) , which is
Revised: i ( t )) , which is

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where D =

Original: [Equation]
Revised: ( A ⊤ Σ − 1 y A + Σ − 1 x ) − 1 A ⊤ Σ − 1 y and γ 2 ry = µ ⊤∗ ( A ⊤ Σ − 1 y A + Σ − 1 x ) − 1 µ ∗ + γ 2 r .

Original: Based on the conditional distribution in (3), in order to maximize the expected reward given the observation, we consider the conditional expectation of the reward given the observations, y i ( t ) ⊤ D ⊤ µ ∗ .
Revised: Based on the conditional distribution in (3), in order to maximize the expected reward given the observations, one can consider the conditional expectation of the reward given the observation; y i ( t ) ⊤ D ⊤ µ ∗ .

Original: So, letting η ∗ =
Revised: So, letting η ∗ =

Original: D ⊤ µ ∗ be the transformed parameter, we focus on the estimation of η ∗ .
Revised: D ⊤ µ ∗ be the transformed parameter, we focus on the estimation of η ∗ .

Original: The rationale is twofold; ﬁrst, the conditional expected reward  can be inferred with only knowing η ∗ , regardless of the exact value of the true parameter µ ∗ .
Revised: The rationale is twofold. First, the conditional expected reward given the observation can be inferred with only learning η ∗ , and one does not need to learn µ ∗ .

Original: Second, µ ∗ is not estimable when the rank of the sensing matrix A in the observation model is less than the dimension of µ ∗ .
Revised: Second, µ ∗ is not estimable when the rank of the sensing matrix A in the observation model is less than d x .

Revised: Therefore, the quality of learning the environment is reﬂected by the estimation accuracy of η ∗ .

Original: Indeed, estimability of µ ∗ needs the restrictive assumptions of nonsingular A and d y ≥ d x .
Revised: Note that consistent estimation of µ ∗ needs further assumptions of non-singularity of A and d y ≥ d x .

Original: The optimal policy that reinforcement learning policies need to compete against knows the true parameter µ ∗ .
Revised: The optimal policy that reinforcement learning policies need to compete against knows the true parameter µ ∗ .

Original: That is, to maximize the reward given theoutput observations, the optimal arm at time t , denoted by a ∗ ( t ) , is
Revised: That is, to maximize the reward given the observations, the optimal arm at time t , denoted by a ∗ ( t ) , is

Original: [Equation]
Revised: [Equation]

Original: Then, the performance degradation due to uncertainty about the environment that the parameter µ ∗ represents, is the assessment criteria for reinforcement learning policies.
Revised: Then, the performance degradation due to uncertainty about the environment that the parameter µ ∗ represents, is the assessment criteria for reinforcement learning policies.

Original: So, we consider the following performance measure, which is commonly used in the literature, and is known as regret of the reinforcement learning policy that selects the sequence of actions a ( t ) , t = 1 , 2 , · · · :
Revised: So, we consider the following performance measure, which is commonly used in the literature, and is known as regret of the reinforcement learning policy that selects the sequence of actions a ( t ) , t = 1 , 2 , · · · :

Original: [Equation]
Revised: [Equation]

Original: In other words, the regret at time T is the total difference in the  obtainedrewards, up to time T ,where the difference at time t is between the optimal arms a ∗ ( t ) and the arm a ( t ) chosen by the reinforcement learning policy based on theoutput observations by the time t .
Revised: In other words, the regret at time T is the total difference in the rewards obtained up to time T , between the optimal arms a ∗ ( t ) and the arm a ( t ) chosen by the reinforcement learning policy based on the observations by the time t .

Original: Note that this difference does not depends on the unknown contexts { x i ( t ) } 1 ≤ i ≤ N . That is, the arm maximizing x i ( t ) ⊤ µ ∗ is not guaranteed to be a ∗ ( t ) , since y i ( t ) ⊤ η ∗ is a realized value of a random variable centered at x i ( t ) ⊤ µ ∗ .
Revised: Note that this difference does not directly depend on the unknown contexts { x i ( t ) } 1 ≤ i ≤ N since the optimal policy that selects a ∗ ( t ) does not observe the context vectors, and decides merely based on the observations y i ( t ) , for i = 1 , · · · , N .

Original: 3. Reinforcement Learning Policy
Revised: 3. Reinforcement Learning Policy

Original: In this section, we explain the details of the Greedy algorithm for contextual bandits with imperfect  observations.
Revised: In this section, we explain the details of the Greedy algorithm for contextual bandits with imperfect context observations.

Original: Although inefﬁcient in some reinforcement learning problems, Greedy algorithms are known to be efﬁcient under certain conditions such as covariate diversity (Bastani et al., 2021).
Revised: Although inefﬁcient in some reinforcement learning problems, Greedy algorithms have a logarithmic regret bound for contextual bandits with fully observed contexts under certain conditions such as covariate diversity (Bastani et al., 2021).

Original: Intuitively, the latter condition expresses that the context vectors cover all directions in R d x with a non-trivial probability, so that additional exploration is not necessary. 
Revised: Intuitively speaking, the latter condition expresses that the context vectors provide information along all dimensions in R d x with positive probability, so that additional exploration is not necessary. 1: Set B (1) = Σ − 1 , b η (1) = η 2: for t = 1 , 2 , . .

Revised: . , do 3:

Original: As discussed in Section 2, it sufﬁces forthe policy to learn to maximize E [ r i  ( t ) 
Revised: Observe observations y i ( t ) for i = 1 , . . . , N 4: Select arm a ( t ) = arg max 1 ≤ i ≤ N y i

Revised: ( t )

Revised: ⊤ b η ( t ) 5:

Revised: Gain reward r a ( t ) ( t ) =

Revised: x a ( t ) ( t ) ⊤ µ ∗ + ψ a ( t ) ( t ) 6:

Revised: Update B ( t + 1) and b η ( t + 1) by (7) and (8) 7: end for

Revised: As discussed in Section 2, it sufﬁces for the policy to learn the arm i that maximizes E [ r i ( t )

Original: |
Revised: |

Original: y
Revised: y

Original: i ( t )]
Revised: i ( t )]

Original: =
Revised: =

Original: y i ( t ) ⊤ η ∗ .
Revised: y i ( t ) ⊤ η ∗ .

Original: To  estimate  the quantity y i ( t ) ⊤ η ∗ , we use the least-squares estimate
Revised: To that end, we estimate η ∗ using the least-squares estimator b η ( t ) = arg max η t X τ = ( r a

Original: [Equation]
Revised: [Equation]

Original:  in lieu of the truth η ∗ .
Revised: Then, the Greedy policy works with b η ( t ) in lieu of the truth η ∗ .

Original: So, theGreedy algorithm selects the arm a ( t ) at time t , such that a ( t ) =
Revised: So, the algorithm selects the arm a ( t ) at time t , such that a ( t ) =

Original: arg max 1 ≤ i ≤ N y i
Revised: arg max 1 ≤ i ≤ N y i

Original: ( t )
Revised: ( t )

Original: ⊤ b η ( t ) .
Revised: ⊤ b η ( t ) .

Original: The recursions to update the parameter estimate b η ( t ) and the empirical inverse covariance matrix B ( t )based on (6) are as follows:
Revised: Thanks to the structure of the parameter estimates in (6), once can update b η ( t ) in a recursive fashion. The recursions relies on updating the empirical inverse covariance matrix B ( t ) as well, according to

Original: [Equation]
Revised: [Equation]

Revised: and

Original: [Equation]
Revised: [Equation]

Original: where the initial values consist of B (1)
Revised: where the initial values consist of B (1)

Original: = Σ − 1 , for some arbitrary postitive deﬁnite matrix Σ , and b η(1) = η for an ar- 1: Set B (1) = Σ − 1 , b η (1) =
Revised: = Σ − 1 , for some arbitrary positive deﬁnite matrix Σ , and b η

Original: η 2: for t = 1 , 2 , . .

Original: . , do 3:

Original: Select arm a

Original: [Equation]
Revised: [Equation]

Original: t
Revised: =

Original: [Equation] Gain reward r a ( t ) ( t ) = x a ( t ) ( t ) ⊤ µ ∗ + ψ a ( t ) ( t ) 5: Update B ( t + 1) and b η( t + 1) by (7) and (8) 6: end for bitrary vector η in R d y .
Revised: η for an arbitrary vector η in R d y .

Original: Algorithm 1 describes the pseudocode for the Greedy algorithm.
Revised: Algorithm 1 describes the pseudocode for the Greedy policy.

Original: 4. Theoretical Performance Guarantees
Revised: 4. Theoretical Performance Guarantees

Original: In this section, we present a theoretical result for Algorithm 1 presented in the previous section.
Revised: In this section, we present a theoretical result for Algorithm 1 presented in the previous section.

Original: The result provides a worst-case analysis and establishes a high probability upper-bound for the regret  in (5).
Revised: The result provides a worst-case analysis and establishes a high probability upper-bound for the regret as deﬁned in (5).

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Assume that Algorithm 1 is used in a bandit with N arms and the output dimension d y .
Revised: Assume that Algorithm 1 is used in a bandit with N arms and the observation dimension d y .

Original: Then, with probability at least 1 − 4 δ , Regret( T ) is of the order
Revised: Then, with probability at least 1 − 4 δ , Regret( T ) is of the order

Original: [Equation]
Revised: [Equation]

Original: where Σ A =
Revised: where cond denotes the condition number of a matrix and γ

Original: A Σ x A ⊤ . 
Revised: [Equation] is the conditional variance of reward deﬁned in ( 3) . The proof of Theorem 4.1 is provided in the appendix. Next, we discuss the intuitions of different terms in the regret bound presented in the above theorem.

Original: The regret bound above scales linearly with the number of arms N ,  with d 3 / 2 y for the dimension of the observations d y , and poly-logarithmically with time  T .
Revised: The regret bound above scales linearly with the number of arms N , and with d 3 / 2 y for the dimension of the observations d y , while it grows poly-logarithmically with time horizon T .

Original: The dimension of  unobserved context vectors do not affect the  regret  because the optimal policy in (4) does not have the exact values of the context vectors.
Revised: The dimension of the unobserved context vectors does not appear in the above regret bound because the optimal policy in (4) does not observe the exact values of the context vectors.

Original: So, similar to the reinforcement learning policy, the optimal policy needs to estimate the contexts as well, as y i ( t )
Revised: So, similar to Algorithm 1, the optimal policy needs to estimate the context vectors as well.

Original: ⊤ η ∗ in (4) is an estimate of x i ( t ) ⊤ µ ∗ for the optimal policy to ﬁnd the optimal arm. The rationale of the linear growth of the regret  with  N is that a policy is more likely to choose one of  sub-optimal arms, when more sub-optimal arms exist, incurring more regret.
Revised: The rationale of the linear growth of the regret bound in Theorem 4.1 with the number of arms N is that for larger N , the policy is more likely to choose one of the sub-optimal arms, which makes the larger growths of the regret more likely.

Original: In addition, the term d 3 / 2 y andthe maximum eigenvalue λ max (Σ A  ) are generated by theuse of truncation for the ℓ 2 norm of vector{+the stochastic observation vectors, for which the following high probability upper-bounds hold; k y i ( t ) k 2 
+}
Revised: In addition, the terms d 3 / 2 y and λ max ( A Σ x A ⊤ + Σ y constituting cond( A Σ x A ⊤ + Σ y ) are generated by the ℓ 2 norm of {+the stochastic observation vectors, for which the following high probability upper-bounds hold; k y i ( t ) k 2 
+}

Original: [Equation]
Revised: = O ( λ max ( A Σ x A ⊤ + Σ y )

Original: as well as the matrix Azuma’s inequality. Further, the polylogarithmic terms of T, N , d y and δ , (log( Nd y T/δ )) 5 /and log( d y T/δ ) , are originated in the truncation event and the Azuma’s inequality.
Revised: d y log( NTd y /δ )) . Similarly, the poly-logarithmic scaling of the regret bound in terms of T and δ , originates from the magnitude of the random vectors ( v T ( δ ) = O ((log( NTd y /δ )) 1 / 2 ) ) that are used in the analysis of Algorithm 1.

Original: Finally, the minimum eigenvalue λ min (Σ A ) and the conditional reward variance γ2 ry are associated with the variance of the estimator b η ( t ) , whose larger value causes a greater regret.
Revised: On the other hand, γ ry indicates the role of the reward noise. Simply, if the reward observations are noisier, it will be harder to learn the optimal arms. Finally, the factor λ min ( A Σ x A ⊤ +Σ y

Original:  The proof of Theorem 4.1 is delegated to Appendices.
Revised: consisting of cond( A Σ x A ⊤ +Σ y in the regret bound is the intrinsic decrease in the cumulative reward caused by uncertainties about the unknown parameter µ ∗ . The proof process is divided into two steps. The ﬁrst step consists of establishing a high probability upper-bound for ( y a ∗ ( t ) ( t ) − y a ( t ) ( t ))

Original: [Figure]
Revised: ⊤ η ∗ .

Original: t
Revised: Then, in the second step we ﬁnd upper-bounds on the frequency of pulling sub-optimal arms. Consequently, by combining the above-mentioned bounds, we prove Theorem 4.1.

Original: 5. Numerical Illustrations
Revised: 5. Numerical Illustrations

Original: In this section, we perform numerical analyses for the theoretical result in the previous section.
Revised: In this section, we perform numerical analyses for the theoretical result in the previous section.

Original: We simulate cases for N = 10 , 20 , 50  and different dimensions of the observations d y = 5 , 20 , 50 with a ﬁxed context dimension d x = 20 .
Revised: We simulate cases with N = 10 , 20 , 50 arms and different dimensions of the observations d y = 5 , 20 , 50 , while the context dimensions are ﬁxed to d x = 20 .

Original: Each case is repeated 100 times and the average and worst quantities of 100 scenarios are reported.
Revised: Each case is repeated 100 times, and the average values of the quantities of interest acrossscenarios are reported, as well as the worst among all the repetitions.

Original: For Figure 1, the left plot depicts the average (solid) and worst-case (dashed) regret among all scenarios, normalized by log t .
Revised: In Figure 1, the left plot depicts the average (solid) and worst-case (dashed) regret among all scenarios, normalized by log t .

Original: The number of arms N varies as shown in the graph, while the  dimension is ﬁxed to d y = 10 .
Revised: The number of arms N varies as shown in the graph, while the observation dimension is ﬁxed to d y = 10 .

Original: Next, the  right one illustrates that the normalized regrets increase over time for different d y at the ﬁxed number of arms N = 5 .
Revised: Next, the graph on the right hand side illustrates that the normalized regrets increase over time for different d y , for the ﬁxed number of arms N = 5 .

Original: For both plots, the worst-case regret curves are well above the average ones, butthe slopes of curves for both cases become ﬂat as time goes on, implying that the worst-case regret grows logarithmically in terms of t as well.
Revised: In Figure 1, the worst-case regret curves are well above the average ones as expected, but curves for both average-case and worst-case become ﬂat as time goes on, implying that the worst-case regret grows logarithmically in terms of t .

Original: Figure 2 presents the average and worst-case regret (non-normalized) at time T = 2000 for different  N = 10 , 20 , 50 and d y = 5 , 20 , 50 .
Revised: Figure 2 presents the average and worst-case regret (not normalized by log T ) at time T = 2000 for different values of N = 10 , 20 , 50 and d y = 5 , 20 , 50 .

Original: The plot shows that the regret at T = 2000 increases as N and d y become larger.
Revised: The plot shows that the regret at T = 2000 increases as N and d y become larger.

Original: In addition, it shows that the dimension of observations d y has a greater effect on the regret than that of the number of arms N .
Revised: In addition, it shows that the dimension of observations d y has a greater effect on the regret than that of the number of arms N , which is consistent with the result of

Revised: Theorem 4.1.

Original: 6. Conclusion
Revised: 6. Conclusion

Original: This work investigates reinforcement learning algorithms for contextual bandits where the contexts are observed imperfectly focusing on the theoretical results about the regret growth. We establish a high probability regret bound for 
Revised: This work investigates reinforcement learning algorithms for contextual bandits where the contexts are observed imperfectly. We focus on establishing theoretical results for regret analysis, and establish a high probability regret bound for Greedy algorithms.

Original: Greedy algorithms, which grows poly-logarithmically with[Figure] the  horizon T .
Revised: The presented regret bound grows poly-logarithmically with the time horizon T .

Original: There are multiple interesting future directions introduced in this paper.
Revised: There exist multiple interesting future directions introduced in this paper.

Original: First, it will be of interest to study reinforcement learning policies for settings that each arm has its own parameter.
Revised: First, it will be of interest to study reinforcement learning policies for settings that each arm has its own parameter.

Original: Further, regret analysis for contextual bandits under imperfect context observations where theother parameters such as the covariance matrices of contexts and observations and the sensing matrix are unknown, is another problem for future work.
Revised: Further, regret analysis for contextual bandits under imperfect context observations where the covariance matrices of context vectors and the sensing matrix are unknown, is another problem for future work.

Original: A. Appendices
Revised: A. Proofs of the Technical Results

Original:  C ( A ) and C ( A ) ⊥ are employed to denote the column-space of the matrix A and its orthogonal subspace, respectively.
Revised: In the sequel, C ( A ) and C ( A ) ⊥ are employed to denote the column-space of the matrix A and its orthogonal subspace, respectively.

Original: In addition, P C ( A ) is the projection operator onto C ( A ) .
Revised: In addition, P C ( A ) is the projection operator onto C ( A ) .

Original: A.1. Proof of Theorem 4. 
Revised: A.1. Proof of Theorem 4.

Original: Proof.
Revised: Proof.

Original: We use the following intermediate results, whose proofs are delegated to Section A.2.
Revised: We use the following intermediate results, whose proofs are delegated to Section A.2.

Original: For simplicity, let b η (1) be a random variable with E [ [Equation]
Revised: For simplicity, let b η (1) be a random variable with E [ b η (1)]

Original:  and 
Revised: = η ∗ and Cov( b

Original: [Equation]
Revised: [Equation]

Original: γ 2 ry so that E [ b η ( t )]
Revised: 2 ry so that E [ b [Equation]

Original: = η ∗ and Cov(b η ( t )
Revised: [Equation] and Cov(

Original: | B

Original: [Equation]
Revised: [Equation]

Original: t ))

Original: =
Revised: =

Original: B ( t ) − 1 γ 2 ry for all t .
Revised: B ( t [Equation] − 1 γ 2 ry for all t .

Original: First, for T > 0 and 0 < δ < 0 . 25 , we deﬁne
Revised: First, for T > 0 and 0 < δ < 0 . 25 , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where v T ( δ ) =
Revised: where v T ( δ ) =

Original: (2 log( Nd y T/δ )) 1 / 2 .
Revised: (2 log( Nd y T/δ )) 1 / 2 .

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For the event W T deﬁned in (9) , we have P ( W T ) ≥ 1 − δ .
Revised: For the event W T deﬁned in (9) , we have P ( W T ) ≥ 1 − δ .

Original: Lemma A.1 guarantees that all the observation up to time T are generated in the truncation event W T with the probability at least 1 − δ .
Revised: Lemma A.1 guarantees that all the observation up to time T are generated in the truncation event W T with the probability at least 1 − δ .

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Let σ { X 1 , . . .
Revised: Let σ { X 1 , . . .

Original: X n } be the sigma-ﬁeld generated by random vectors X 1 , . . .
Revised: X n } be the sigma-ﬁeld generated by random vectors X 1 , . . .

Original: , X n .
Revised: , X n .

Original: For the observation of chosen arm y a ( t ) ( t ) at time t, the estimator b η ( t ) deﬁned in (8) , and the ﬁltration { F t } 1 ≤ t ≤ T deﬁned according to
Revised: For the observation of chosen arm y a ( t ) ( t ) at time t, the estimator b η ( t ) deﬁned in (8) , and the ﬁltration { F t } 1 ≤ t ≤ T deﬁned according to

Original: [Equation]
Revised: [Equation]

Original:  we have
Revised: { r a ( τ ) ( τ ) } 1 ≤ τ ≤ t } , we have

Original: [Equation]
Revised: [Equation]

Original: where V t =
Revised: where V t =

Original: S − 1 / 2 y 
Revised: S − 1 / 2 y y a

Original: [Equation]
Revised: [Equation]

Original: and k N = E
Revised: y a [Equation] −

Original: [Equation]
Revised: [Equation]

Original:  # for N independent Z i with the standard normal distribution and S y =
Revised: y and k N = E "(cid:18) max 1 ≤ i ≤ N { Z i } (cid:19) # for N independent Z i with the standard normal distribution and S y =

Original: Cov( y i ( t )) .
Revised: Cov( y i ( t )) .

Original: That is, k N is the expected maximum of N independent standard normal random variables.
Revised: That is, k N is the expected maximum of N independent standard normal random variables.

Original: Lemma A.2 sets the stage for analysis of the (unnormalized) empirical inverse covariance B ( t ) in (7)
Revised: Lemma A.2 sets the stage for analysis of the (unnormalized) empirical inverse covariance B ( t ) in (7)

Original: Lemma A.3.
Revised: Lemma A.3.

Original: (Matrix Azuma Inequality (Tropp, 2012)) Consider the sequence { M k } 1 ≤ k ≤ K of symmetric d × d random matrices adapted to some ﬁltration { G k } 1 ≤ k ≤ K , such that E
Revised: (Matrix Azuma Inequality (Tropp, 2012)) Consider the sequence { M k } 1 ≤ k ≤ K of symmetric d × d random matrices adapted to some ﬁltration { G k } 1 ≤ k ≤ K , such that E

Original: [ M k | G k − 1 ] = 0 .
Revised: [ M k | G k − 1 ] = 0 .

Original: Assume that there is a deterministic sequence of symmetric matrices { A k } 1 ≤ k ≤ K that satisfy M 2 k (cid:22)
Revised: Assume that there is a deterministic sequence of symmetric matrices { A k } 1 ≤ k ≤ K that satisfy M 2 k (cid:22)

Original: A 2 k , almost surely.
Revised: A 2 k , almost surely.

Original: Let σ 2 = k
Revised: Let σ 2 = k

Original: P 1 ≤ k ≤ K A 2 k k .
Revised: P 1 ≤ k ≤ K A 2 k k .

Original: Then, for all ε ≥ 0 , it holds that
Revised: Then, for all ε ≥ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Lemma A.4 provides a high probability lower bound for the minimum eigenvalue of B ( t ) .
Revised: Lemma A.4 provides a high probability lower bound for the minimum eigenvalue of B ( t ) .

Original: Then, Lemma A.5 bounds the estimation error.
Revised: Then, Lemma A.5 bounds the estimation error.

Original: Lemma A.4.
Revised: Lemma A.4.

Original: For B ( t ) in (7) and t ≤ T , on the event W T deﬁned in (9) , by Lemma A.2 and A.3, with the probability at least 1 − δ , we have
Revised: For B ( t ) in (7) and t ≤ T , on the event W T deﬁned in (9) , by Lemma A.2 and A.3, with the probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma A.5.
Revised: Lemma A.5.

Original: In Algorithm 1, let b η ( t ) be the parameter estimate, as deﬁned in (8) .
Revised: In Algorithm 1, let b η ( t ) be the parameter estimate, as deﬁned in (8) .

Original: Then, for t ≤ T , on the event W T deﬁned (9) , we have
Revised: Then, for t ≤ T , on the event W T deﬁned (9) , we have

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Next, Lemma A.6 gives an upper bound for the probability that Algorithm 1 does not choose the optimal arm at time t .
Revised: Next, Lemma A.6 gives an upper bound for the probability that Algorithm 1 does not choose the optimal arm at time t .

Original: Finally, Lemma A.7 studies the weighted sum of indicator functions
Revised: Finally, Lemma A.7 studies the weighted sum of indicator functions

Original: I ( a ∗ ( t ) 6 = a ( t )) that count the effective number of times that the algorithm chooses sub-optimal arms.
Revised: I ( a ∗ ( t ) 6 = a ( t )) that count the effective number of times that the algorithm chooses sub-optimal arms.

Original: Lemma A.6.
Revised: Lemma A.6.

Original: Given B ( t ) , an upper bound of probability of choosing a sub-optimal arm is bounded as follows:
Revised: Given B ( t ) , an upper bound of probability of choosing a sub-optimal arm is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ t = λ max ( B ( t ) − 1 ) .
Revised: where λ t = λ max ( B ( t ) − 1 ) .

Original: Lemma A.7.
Revised: Lemma A.7.

Original: For I ( a ∗ ( t ) 6 = a ( t )) , on the event W T , with the probability at least 1 − δ , we have
Revised: For I ( a ∗ ( t ) 6 = a ( t )) , on the event W T , with the probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where t ∗ = 128 v T ( δ ) 4 log d y Tδ + 1 .
Revised: where t ∗ = 128 v T ( δ ) 4 log d y Tδ + 1 .

Original: Note that Regret( T ) is the sum of the conditional expected reward difference
Revised: Note that Regret( T ) is the sum of the conditional expected reward difference

Original: [Equation]
Revised: [Equation]

Original: The difference (cid:0) y a [Equation] [Equation]
Revised: for 1 ≤ t ≤ T .

Original: [Equation] t [Equation] −
Revised: The difference

Original: [Equation]
Revised: [Equation]

Original: t ) [Equation] t ) (cid:1) ⊤ η ∗ at time t is greater than 0, only when a ∗ ( t ) 6 = a ( t ) .
Revised: ⊤ η ∗ at time t is greater than 0, only when a ∗ ( t ) 6 = a ( t ) .

Original: Thus, the regret can be rewritten as Regret( T ) =
Revised: Thus, the regret can be rewritten as Regret( T ) =

Original: [Equation]
Revised: [Equation]

Original: ⊤ η ∗ I

Original: [Equation]

Original: a ∗ ( t ) 6 = a ( t )) .
Revised: ( t ) 6 = a ( t )) .

Original: To ﬁnd an upper bound of the regret, we ﬁnd high probability upper bounds for(cid:0)
Revised: To ﬁnd an upper bound of the regret, we ﬁnd high probability upper bounds for

Original: [Equation]
Revised: [Equation]

Original: (cid:1)
Revised: and I

Original: [Equation]
Revised: [Equation]

Original: η ∗ and I ( a ∗ ( t ) 6 = a ( t )) , respectively.
Revised: a [Equation] , respectively.

Original: For both upper bounds, the inverse of the (unnormalized) empirical covariance matrix B ( t ) in (7) matters in that the matrix determines the size of estimation error k
Revised: For both upper bounds, the inverse of the (unnormalized) empirical covariance matrix B ( t ) in (7) matters in that the matrix determines the size of estimation error k

Original: [Equation]
Revised: [Equation]

Original: By, Lemma A.4, we have
Revised: By, Lemma A.4, we have

Original: [Equation]
Revised: [Equation]

Original: for all 1 ≤ t ≤ T with the probability at least 1 − 2 δ .
Revised: for all 1 ≤ t ≤ T with the probability at least 1 − 2 δ .

Original: This implies that B ( t ) grows linearly with the horizon almost surely.
Revised: This implies that B ( t ) grows linearly with the horizon almost surely.

Original: Next, we investigate the estimation error k η ∗ − b η ( t ) k based on the above result of the minimum eigenvalue of B ( t ) .
Revised: Next, we investigate the estimation error k η ∗ − b η ( t ) k based on the above result of the minimum eigenvalue of B ( t ) .

Original: Using k y i ( t ) k ∞ ≤ λ max ( S y ) 1 / 2 v T ( δ ) on the event W T , we have
Revised: Using k y i ( t ) k ∞ ≤ λ max ( S y ) 1 / 2 v T ( δ ) on the event W T , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where λ max ( S y ) =
Revised: where λ max ( S y ) =

Original: λ max ( S y ) .
Revised: λ max ( S y ) .

Original: So, we write the regret in the following form:
Revised: So, we write the regret in the following form:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, we denote λ t = λ max ( B ( t ) − 1 ) =
Revised: Here, we denote λ t = λ max ( B ( t ) − 1 ) =

Original: ( λ min ( B ( t ))) − 1 .
Revised: ( λ min ( B ( t ))) − 1 .

Original: By (10), we can ﬁnd t ∗ =
Revised: By (10), we can ﬁnd t ∗ =

Original: 128 v T ( δ ) 4 log
Revised: 128 v T ( δ ) 4 log

Original: d
Revised: d

Original: y Tδ
Revised: y Tδ

Original: + 1 , such that
Revised: + 1 , such that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 13) with the probability at least 1 − δ , for all t ∗ < t ≤ T .
Revised: (13) with the probability at least 1 − δ , for all t ∗ < t ≤ T .

Original: By Lemma A.5 and (13), for all t ∗ < t ≤ T , with the probability at least 1 − 3 δ , we have
Revised: By Lemma A.5 and (13), for all t ∗ < t ≤ T , with the probability at least 1 − 3 δ , we have

Original: [Equation]
Revised: [Equation]

Original: 14) where a=4( λ max ( S
Revised: 14) where a=

Original: [Equation]
Revised: [Equation]

Original: /λ
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( S y
Revised: ( δ

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: T ( δ ) p 2 d y log(

Revised: p 2 d y log(

Original: Tδ − 1 ) .
Revised: Tδ − 1 ) .

Original: Thus, with ( y a ∗ ( t ) − y a ( t ) )
Revised: Thus, with ( y a ∗ ( t ) − y a ( t ) )

Original: ⊤ η ∗ ≤ 2 λ max ( S y ) 1 / 2 v T ( δ ) k
Revised: ⊤ η ∗ ≤ 2 λ max ( S y ) 1 / 2 v T ( δ ) k

Original: η ∗ k for t < t ∗ , the regret can be represented
Revised: η ∗ k for t < t ∗ , the regret can be represented

Original: [Equation]
Revised: [Equation]

Original: with the probability at least 1 − 3 δ .
Revised: with the probability at least 1 − 3 δ .

Original: Now, we consider the probability to choose the optimal arm at time t .
Revised: Now [Equation] we consider the probability to choose the optimal arm at time t .

Original: By Lemma A.6, we have
Revised: By Lemma A.6, we have

Original: [Equation]
Revised: [Equation]

Original: log T
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: Now, we construct an upper bound about the indicator function I ( a ∗ ( t ) 6 = a ( t )) in (12), by Lemma A.7.
Revised: we construct an upper bound about the indicator function I ( a ∗ ( t ) 6 = a ( t )) in (12), by Lemma A.7.

Original: [Equation]
Revised: [Equation]

Original: 17) with the probability at least 1 − δ .
Revised: with the probability at least 1 − δ .

Original: Therefore, by (16) and (17), with the probability at least 1 − 4 δ , the following inequalities hold for the regret of the algorithm, which yield to the desired result:
Revised: Therefore, by (16) and (17) [Equation] with the probability at least 1 − 4 δ , the following inequalities hold for the regret of the algorithm, which yield to the desired result:

Original: [Equation]
Revised: [Equation]

Original: Finally, using S y =
Revised: Finally, using S y =

Original: A Σ x A ⊤ + Σ y and λ max ( S y )
Revised: A Σ x A ⊤ + Σ y and λ max ( S y )

Original: /λ min ( S y )
Revised: /λ min ( S y )

Original: =
Revised: =

Original: O (( λ max (Σ A ) + λ max (Σ y )) / ( λ min (Σ A ) + λ min (Σ y ))) , with the probability at least 1 − 4 δ , we haveRegret
Revised: O (( λ max (Σ A ) + λ max (Σ y )) / ( λ min (Σ A ) + λ min (Σ y ))) , with the probability at least 1 − 4 δ , we have

Original: [Equation]
Revised: [Equation]

Original: A.2. Proofs of Lemmas A.2.1. P ROOF OF L EMMA A. 
Revised: A.2. Proofs of Lemmas A.2.1. P ROOF OF L EMMA A.

Original: Note that S − 0 .
Revised: Note that S − 0 .

Original: y y i ( t ) has the normal distribution N (0 , I d y ) .
Revised: y y i ( t ) has the normal distribution N (0 , I d y ) .

Original: Then, we have
Revised: Then, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where y ij ( t ) is the j th component of y i ( t ) .
Revised: where y ij ( t ) is the j th component of y i ( t ) .

Original: By plugging v T ( δ ) to ε , we have
Revised: By plugging v T ( δ ) to ε , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: A.2.2. P ROOF OF L EMMA A. 
Revised: A.2.2. P ROOF OF L EMMA A.

Original: We use the following decomposition
Revised: We use the following decomposition

Original: [Equation]
Revised: [Equation]

Original: We claim that P C ( S 0 .
Revised: We claim that P C ( S 0 .

Original: 5 y b η ( t ))
Revised: 5 y b [Equation] ( t ))

Original: S − 0 .
Revised: S − [Equation]

Original: 5 y y a ( t ) and P C ( S 0 .
Revised: y y a ( t ) and P C ( S 0 .

Original: 5 y b η ( t )) ⊥ S − 0 .
Revised: 5 y b η ( t )) ⊥ S − 0 .

Original: 5 y
Revised: y y

Original: [Equation]

Original: i ( t ) are statistically independent.
Revised: i ( t ) are statistically independent.

Original: To show it, deﬁne
Revised: To show it, deﬁne

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where Z i has the distribution N ( 0 d y , I d y ) and ν is an arbitrary vector in R d y .
Revised: where Z i has the distribution N ( 0 d y , I d y ) and ν is an arbitrary vector in R d y .

Original: The vector Z i can be decomposed as
Revised: The vector Z i can be decomposed as

Original: Z i
Revised: Z i

Original: = P C ( ν )
Revised: = P C ( ν )

Original: Z i +( I d − P C
Revised: Z i +( I d − P C

Original: ( ν ) )
Revised: ( ν ) )

Original: Z i .
Revised: Z i .

Original: Then, we have Z ( ν, N ) =
Revised: Then, we have Z ( ν, [Equation] =

Original: arg max Z i , 1 ≤ i ≤ N(cid:8) ( P C ( ν ) Z i )
Revised: [Equation] i , [Equation] ≤ i ≤ N

Original: ⊤ ν (cid:9) , because P C ( ν ) ν
Revised: [Equation] (cid:9) , because P C ( ν ) ν

Original: = ν .
Revised: = ν .

Original: This implies that only the ﬁrst term of the decomposed terms, P C ( ν ) Z i , affects the result of arg max Z i , 1 ≤ i ≤ N (cid:8)
Revised: This implies that only the ﬁrst term of the decomposed terms, P C ( ν ) Z i , affects the result of arg max Z i , 1 ≤ i ≤ N (cid:8)

Original: [Equation]
Revised: [Equation]

Original: ⊤ i ν (cid:9) .
Revised: ⊤ i [Equation] (cid:9) .

Original: This means that Z ( ν, N ) has the same distribution as P C ( ν )
Revised: This means that Z ( ν, N ) has the same distribution as P C ( ν )

Original: Z ( ν, N ) + ( I d − P C ( ν ) )
Revised: Z ( ν, N ) + ( I d − P C ( ν ) )

Original: Z i , which means
Revised: Z i , which means

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where d = is used to denote the equality of the probability distributions.
Revised: where d = is used to denote the equality of the probability distributions.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: S − 0 .
Revised: S − 0 .

Original: 5 y y a ( t ) has the same distribution as
Revised: 5 y y a ( t ) has the same distribution as

Original: P

Original: [Equation]
Revised: [Equation]

Original: y i ( t ) , where

Original: P
Revised: y

Original: [Equation]
Revised: [Equation]

Original: y a ( t ) and P C ( S 0 . 5 y b
Revised: ( t ) , where

Original: [Equation]
Revised: [Equation]

Original: ( t
Revised: y a

Original: [Equation]
Revised: [Equation]

Original: S
Revised: t ) and P

Original: [Equation]
Revised: [Equation]

Original: yy i ( t ) are statistically independent.
Revised: y i ( t ) are statistically independent.

Original: By the decomposition (23) and the independence, E(cid:2)
Revised: By the decomposition (23) and the independence, E

Original: [Equation]
Revised: [Equation]

Original: (cid:3) can be written as
Revised: (cid:3) can be written as

Original: E

Original: [Equation]
Revised: [Equation]

Original: To proceed, we show that the ﬁrst term above, E [ P S0 .
Revised: To proceed, we show that the ﬁrst term above, E [ P S

Original: [Equation]
Revised: [Equation]

Original: y b η ( t) S [Equation] [Equation] η
Revised: t

Original: [Equation]

Original: ] = aP

Original: [Equation]

Original: η

Original: [Equation]
Revised: [Equation]

Original: t
Revised: S

Original: [Equation]
Revised: [Equation]

Original: for some constant a > 1 .
Revised: for some constant a > 1 .

Original: Using P C ( ν ) = νν ⊤ /ν ⊤ ν for an arbitrary vector ν ∈ R d
Revised: Using P C ( ν ) = νν ⊤ /ν ⊤ ν for an arbitrary vector ν ∈ R d

Original: y , we have
Revised: y , we have

Original: [Equation]
Revised: [Equation]

Original: 27) where −−−−−→ S 0 .
Revised: [Equation] where −−−−−→ S 0 .

Original: 5 y 
Revised: 5 y b η

Original: [Equation]
Revised: t ) = S 0 . 5 y b η ( t ) / k S 0 .

Original:  b η [Equation]
Revised: 5 y b η ( t ))

Original: k is the unit vector aligned linearly with S 0 .
Revised: k is the unit vector aligned linearly with S 0 .

Original: 5 y b η
Revised: 5 y b η

Original: t ) .
Revised: t ) .

Original: Now, it sufﬁces to prove that
Revised: Now, it sufﬁces to prove that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original:  t ) has the standard normal distribution, since S − 0 .
Revised: y i [Equation] t ) has the standard normal distribution, since S − 0 .

Original:  y y
Revised: 5 y y

Original: i
Revised: i

Original: ( t ) has the distribution N (0 , I d y ) .
Revised: [Equation] t ) has the distribution N (0 , I d y ) .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  ) is the maximum variable of N variables with the standard normal density.
Revised: t ) is the maximum variable of N variables with the standard normal density.

Original: Thus, using
Revised: Thus, using

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where d = denotes the equality in terms of distribution.
Revised: where d = denotes the equality in terms of distribution.

Original: As such, we haveE
Revised: As such [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the quantity in (29) as k N ,
Revised: We deﬁne the quantity in (29) as k N ,

Original: [Equation]
Revised: [Equation]

Original: which is greater than 1 for N ≥ 2 and grows as N gets larger, because E [ V 
Revised: which is greater than 1 for N ≥ 2 and grows as N gets larger [Equation] because E [ V 2 i ] =

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: E [

Original: [Equation]
Revised: [Equation]

Original: A.2.3. P ROOF OF L EMMA A. 
Revised: A.2.3. P ROOF OF L EMMA A.

Original: Consider Vt =
Revised: Consider V

Original: S − 1 / 2 y y a ( t ) ( t )
Revised: [Equation]

Original: y a ( t ) ( t ) ⊤ S − 1 / 2 y deﬁned in Lemma A.2 to identify the behavior of B ( t ) .
Revised: deﬁned in Lemma A.2 to identify the behavior of B ( t ) .

Original: By Lemma A.2, the minimum eigenvalue of E [ V t | F t − 1 ] is greater than 1 for all t .
Revised: By Lemma A.2, the minimum eigenvalue of E [ V t | F t − 1 ] is greater than 1 for all t .

Original: Thus, for all t > 0 , it holds that
Revised: Thus, for all t > 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: min

Original: [Equation]
Revised: [Equation]

Original: Now, we focus on a high probability lower-bound for the smallest eigenvalue of B ( t ) .
Revised: Now, we focus on a high probability lower-bound for the smallest eigenvalue of B ( t ) .

Original: On the event W T , the matrix v 2 T ( δ ) I − V t is positive semideﬁnite for all i and t .
Revised: On the event W T , the matrix v 2 T ( δ ) I − V t is positive semideﬁnite for all i and t .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then, X τ =
Revised: Then, X τ =

Original: Y τ − Y τ − 1 and E
Revised: Y τ − Y τ − 1 and E

Original: [ X τ | F τ − 1 ] = 0 .
Revised: [ X τ | F τ − 1 ] = 0 .

Original: Thus, X τ is a martingale difference sequence.
Revised: Thus, X τ is a martingale difference sequence.

Original: Because v 2 T ( δ ) I − V t (cid:23)for all t ≤ T , 4 v 4 T ( δ ) I − X 2 τ (cid:23) 0 , for all τ ≤ T , on the event W T .
Revised: Because v 2 T ( δ ) I − V t (cid:23)for all t ≤ T , 4 v 4 T ( δ ) I − X 2 τ (cid:23) 0 , for all τ ≤ T , on the event W T .

Original: By Lemma A.3, we get
Revised: By Lemma A.3, we get

Revised: P λ min

Original: [Equation]
Revised: [Equation]

Original: (34) for ε ≤ 0 .
Revised: (34) for ε ≤ 0 .

Original: Now, using
Revised: Now, using

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: P

Original: [Equation]

Original: P t [Equation] 1 τ =1 E [ V τ | F τ − 1 ] , together withλ min t −
Revised: ] , together with

Original: [Equation]

Original: E [

Original: [Equation]
Revised: [Equation]

Original:  and (32), we obtain
Revised: (35) and (32), we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where − 1 ≤ ε ≤ 0 is arbitrary, and we used the fact that λ min [Equation] ≥ 0 .
Revised: where − 1 ≤ ε ≤ 0 is arbitrary, and we used the fact that λ min (cid:16)P t − 1 τ =1 V τ (cid:17) ≥ 0 .

Original: Indeed, using
Revised: Indeed, using

Original: P [Equation] [Equation] =
Revised: P t − 1 τ =1 V τ =

Original: S − 0 .
Revised: S − 0 .

Original: 5 y
Revised: 5 y

Original: B ( t ) S − 0 .
Revised: B ( t ) S − 0 .

Original: 5 y , on the event W T deﬁned in (9), for − 1 ≤ ε ≤ 0 we have 
Revised: 5 y , on the event W T deﬁned in (9), for − 1 ≤ ε ≤ 0 we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: where λ min ( S y )
Revised: where λ min ( S y )

Original: =
Revised: =

Original: λ min ( S y ) .
Revised: λ min ( S y ) .

Original: In other words, by equating d y · exp
Revised: In other words, by equating d y · exp

Original: [Equation]
Revised: [Equation]

Original: (cid:1) to δ/T ,(37) can be written as
Revised: to δ/T ,

Original: [Equation]
Revised: [Equation]

Revised: 37) can be written as

Original: [Equation]
Revised: [Equation]

Original: for all 1 ≤ t ≤ T with the probability at least 1 − 2 δ .
Revised: for all 1 ≤ t ≤ T with the probability at least 1 − 2 δ .

Original: A.2.4. P ROOF OF L EMMA A. 
Revised: A.2.4. P ROOF OF L EMMA A.

Original: Note that b η ( t ) has the distribution N ( E [ 
Revised: Note that b η ( t ) has the distribution N ( E [ [Equation]

Original: [Equation]
Revised: [Equation]

Original: )) given the observations up to time t , where
Revised: )) given the observations up to time t , where

Original: [Equation]
Revised: [Equation]

Original: For Z ∼ N (0 , λ max ( B ( t ) − 1 ) γ 2 ry ) , using the Chernoff bound, we get
Revised: For Z ∼ N (0 , λ max ( B ( t ) − 1 ) γ 2 ry ) , using the Chernoff bound, we get

Original: [Equation]
Revised: [Equation]

Original:  where ε ≥ 0 .
Revised: 40) where ε ≥ 0 .

Original: A.2.5. P ROOF OF L EMMA A. 
Revised: A.2.5. P ROOF OF L EMMA A.

Original: Let a ∗∗ ( t ) be the arm with the second largest expected reward at time t and η ∗∗ be a vector such
Revised: Let a ∗∗ ( t ) be the arm with the second largest expected reward at time t and η ∗∗ be a vector such

Original: that y a ∗ ( t ) ( t ) ⊤
Revised: that y a ∗ ( t ) ( t ) ⊤

Original: η
Revised: η

Original: ∗∗ = y a ∗∗ ( t ) ( t ) ⊤ η ∗∗ and θ ( y a ∗ ( t ) ( t ) − y a ∗∗ ( t ) ( t ) , η ∗ − η ∗∗ ) = 0 , where θ ( x, y ) is the angle between two vectors x and y .
Revised: ∗∗ = y a ∗∗ ( t ) ( t ) ⊤ η ∗∗ and θ ( y a ∗ ( t ) ( t ) − y a ∗∗ ( t ) ( t ) , η ∗ − η ∗∗ ) = 0 , where θ ( x, y ) is the angle between two vectors x and y .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: If k y a 
Revised: If k y a ∗ ( t ) ( t ) − y a ∗∗ ( t ) ( t ) k k η ∗ − b

Original: [Equation]
Revised: [Equation]

Original: y
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: η
Revised: y

Original: [Equation]
Revised: [Equation]

Original: − b η ( t ) k ≤ ( y a ∗

Original: [Equation]

Original: t ) ( t ) − y a ∗∗ ( t ) ( t ))
Revised: t ) ( t ) − y a ∗∗ ( t ) ( t ))

Original: ⊤ η ∗ , we can guarantee a ∗ ( t )
Revised: ⊤ η ∗ [Equation] we can guarantee a ∗ ( t )

Original: = a ( t ) .
Revised: = a ( t ) .

Original: Thus, the probability not to choose the optimal arm at time t given the observations and B ( t ) is
Revised: Thus, the probability not to choose the optimal arm at time t given the observations and B ( t ) is

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: Using k y a [Equation] [Equation] t ) 
Revised: Using k y a ∗ ( t ) ( t ) − y a ∗∗ ( t ) ( t ) k 2 ≤ 2 λ max (Σ A )

Original: [Equation]

Original: y [Equation] [Equation] t ) k 2 ≤ 2 λ max (Σ A ) d y v T ( δ ) 2 on the event W T , we have
Revised: d y v T ( δ ) 2 on the event W T , we have

Original: [Equation]
Revised: [Equation]

Original: Let X 1 . . .
Revised: Let X 1 . . .

Original: X N be the order statistics of variables with the standard normal density.
Revised: X N be the order statistics of variables with the standard normal density.

Original: The joint distribution of the maximum,
Revised: The joint distribution of the maximum,

Original: X N , and the second maximum variable, X N − 1 , of N independent ones with the standard normal density is 
Revised: X N , and the second maximum variable, X N − 1 , of N independent ones with the standard normal density is [Equation]

Original: [Equation]
Revised: [Equation]

Original: where φ and Φ are the pdf and cdf of the standard normal distribution, respectively.
Revised: where φ and Φ are the pdf and cdf of the standard normal distribution, respectively.

Original: The density of D = X N − X N − 1 , which is the difference of the maximum and second largest variable, can be bounded by Nφ (0) as follows:
Revised: The density of D = X N − X N − 1 , which is the difference of the maximum and second largest variable, can be bounded by Nφ (0) as follows:

Original: [Equation]
Revised: [Equation]

Original: Thus, the density γD is bounded by Nφ (0) /γ =
Revised: Thus, the density γD is bounded by Nφ (0) /γ =

Original: N/ p 2 πγ 2 .
Revised: N/ p 2 πγ 2 .

Original: We denote ∆ t =
Revised: We denote ∆ t =

Original: ( y a ∗ ( t ) ( t ) − y a ∗∗ ( t ) ( t ))
Revised: ( y a ∗ ( t ) ( t ) − y a ∗∗ ( t ) ( t ))

Original: ⊤ η ∗ .
Revised: ⊤ η ∗ .

Original: The term on the right hand side is the upper bound P ( a ∗ ( t ) 6 = a ( t ) | B ( t ) , ∆ t ) .
Revised: The term on the right hand side is the upper bound P ( a ∗ ( t ) 6 = a ( t ) | B ( t ) , ∆ t ) .

Original: Thus, by marginalizing ∆ t from it, we have
Revised: Thus, by marginalizing ∆ t from it, we have

Original: [Equation]
Revised: [Equation]

Original: where the density of ∆ t , f ∆ t (∆ t ) , is bounded by N/ p [Equation] ∗ by (45).
Revised: where the density of ∆ t , f ∆ t (∆ t ) , is bounded by N/ p 2 πη ⊤∗ S y η ∗ by (45).

Original: A.2.6. P ROOF OF L EMMA A. 
Revised: A.2.6. P ROOF OF L EMMA A.

Original: We construct a martingale difference sequence that satisﬁes the conditions in Lemma A.3.
Revised: We construct a martingale difference sequence that satisﬁes the conditions in Lemma A.3.

Original: To that end, let G 1 =
Revised: To that end, let G 1 =

Original: H 1 = 0 ,
Revised: H 1 = 0 ,

Original: G [Equation] [Equation] I
Revised: G τ

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]

Original: −

Original: [Equation]

Original: | F

Original: [Equation] t [Equation] 1 ) , and H t =
Revised: and H t =

Original: P tτ =
Revised: P tτ =

Original: 1 G τ , where
Revised: 1 G τ , where

Original: F ∗ t − 1 = σ {{ B ( τ ) } 1 ≤ τ ≤ t − 1 } .
Revised: [Equation]

Original: Since E [ G τ | F ∗ τ − 1 ] = 0 , the above sequences { G τ } τ ≥ 0 and { H τ } τ ≥ 0 are a martingale difference sequence and a martingale with respect to the ﬁltration { F ∗ τ } 1 ≤ τ ≤ T , respectively.
Revised: Since E [ G τ | F ∗ τ − 1 ] = 0 , the above sequences { G τ } τ ≥ 0 and { H τ } τ ≥ 0 are a martingale difference sequence and a martingale with respect to the ﬁltration { F ∗ τ } 1 ≤ τ ≤ T , respectively.

Original: Let c τ = 2( τ − 1) − 1 / 2 .
Revised: Let c τ = 2( τ − 1) − 1 / 2 .

Original: Since
Revised: Since

Original: P

Original: Tτ =1 | G

Original: [Equation]

Original: | ≤

Original: P

Original: Tτ =

Original: [Equation]
Revised: [Equation]

Original: ≤ 4 log T , by Lemma A.3, we have
Revised: 4 log T , by Lemma A.3, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, with the probability at least 1 − δ , it holds that
Revised: Thus, with the probability at least 1 − δ , it holds that

Original: [Equation]
Revised: [Equation]