Original: Instance-optimal PAC Algorithms for Contextual Bandits
Revised: Instance-optimal PAC Algorithms for Contextual Bandits

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In the stochastic contextual bandit setting, regret-minimizing algorithms havebeen extensively researched, but their instance-minimizing best-arm identiﬁcationcounterparts remain seldom studied.
Revised: In the stochastic contextual bandit setting, regret-minimizing algorithms have been extensively researched, but their instance-minimizing best-arm identiﬁcation counterparts remain seldom studied.

Original: In this work, we focus on the stochastic banditproblem in the ( ✏ ,  ) - PAC setting: given a policy class ⇧ the goal of the learneris to return a policy ⇡ 2 ⇧ whose expected reward is within ✏ of the optimalpolicy with probability greater than 1   .
Revised: In this work, we focus on the stochastic bandit problem in the ( ✏ ,  ) - PAC setting: given a policy class ⇧ the goal of the learner is to return a policy ⇡ 2 ⇧ whose expected reward is within ✏ of the optimal policy with probability greater than 1   .

Original: We characterize the ﬁrst instancedependent PAC sample complexity of contextual bandits through a quantity ⇢ ⇧ ,and provide matching upper and lower bounds in terms of ⇢ ⇧ for the agnostic andlinear contextual best-arm identiﬁcation settings.
Revised: We characterize the ﬁrst instancedependent PAC sample complexity of contextual bandits through a quantity ⇢ ⇧ , and provide matching upper and lower bounds in terms of ⇢ ⇧ for the agnostic and linear contextual best-arm identiﬁcation settings.

Original: We show that no algorithm can besimultaneously minimax-optimal for regret minimization and instance-dependent
Revised: We show that no algorithm can be simultaneously minimax-optimal for regret minimization and instance-dependent

Original: PAC for best-arm identiﬁcation.
Revised: PAC for best-arm identiﬁcation.

Original: Our main result is a new instance-optimal andcomputationally efﬁcient algorithm that relies on a polynomial number of calls toan argmax oracle.
Revised: Our main result is a new instance-optimal and computationally efﬁcient algorithm that relies on a polynomial number of calls to an argmax oracle.

Original:  Introduction
Revised:  Introduction

Original: We consider the stochastic contextual bandit problem in the PAC setting.
Revised: We consider the stochastic contextual bandit problem in the PAC setting.

Original: Fix a distribution ⌫ over apotentially countable 1 set of contexts C .
Revised: Fix a distribution ⌫ over a potentially countable 1 set of contexts C .

Original: The action space is A  and for computational tractability,we assume |A| is ﬁnite.
Revised: The action space is A , and for computational tractability, we assume |A| is ﬁnite.

Original: We have a set of policies ⇧ of interest where each policy ⇡ 2 ⇧ is a map17 from contexts to an action space ⇡ :
Revised: We have a set of policies ⇧ of interest where each policy ⇡ 2 ⇧ is a map from contexts to an action space ⇡ :

Original: C !
Revised: C !

Original: A .
Revised: A .

Original: The reward function is r : C ⇥ A !
Revised: The reward function is r : C ⇥ A !

Original: R .
Revised: R .

Original: At eachtime t = 1 , 2 , . . .
Revised: At each time t = 1 , 2 , . . .

Original: a context c t ⇠ ⌫ arrives, the learner chooses an action a t 2 A , and receivesreward r t :=
Revised: a context c t ⇠ ⌫ arrives, the learner chooses an action a t 2 A , and receives reward r t :=

Original: r t ( c t , a t ) 2 R with E [ r t | c t , a t ] = r ( c t , a t ) 2 R .
Revised: r t ( c t , a t ) 2 R with E [ r t | c t , a t ] = r ( c t , a t ) 2 R .

Original: The value of a policy V ( ⇡ ) is theexpected reward from playing action ⇡ ( c ) in context c : V ( ⇡ ) =
Revised: The value of a policy V ( ⇡ ) is the expected reward from playing action ⇡ ( c ) in context c : V ( ⇡ ) =

Original: E c ⇠ ⌫ [ r ( c, ⇡ ( c ))] .
Revised: E c ⇠ ⌫ [ r ( c, ⇡ ( c ))] .

Original: Given a collectionof policies ⇧ , the objective is to identify the optimal policy ⇡ ⇤ := arg max ⇡ 2 ⇧ V ( ⇡ ) , with highprobability.
Revised: Given a collection of policies ⇧ , the objective is to identify the optimal policy ⇡ ⇤ := arg max ⇡ 2 ⇧ V ( ⇡ ) , with high probability.

Original: Formally, for any ✏ > 0 and  2 (0 , 1) , we seek to characterize the sample complexity ofidentifying a policy ⇡ 2 ⇧ such that V ( ⇡ )  V ( ⇡ ⇤ )
Revised: Formally, for any ✏ > 0 and  2 (0 , 1) , we seek to characterize the sample complexity of identifying a policy ⇡ 2 ⇧ such that V ( ⇡ )  V ( ⇡ ⇤ )

Original:  ✏ , with probability at least 1   .
Revised:  ✏ , with probability at least 1   .

Original: That is, wewish to minimize the total amount of interactions with the environment to learn an ✏ -optimal policy.
Revised: That is, we wish to minimize the total amount of interactions with the environment to learn an ✏ -optimal policy.

Original: We study both the agnostic setting, where ⇧ is an arbitrary set of policies with no assumed relationshipwith the reward function r ( c, a ) ; and the realizable setting, where the policy class and the rewardfunction follow a linear structure, known as the linear contextual bandit problem.
Revised: We study both the agnostic setting, where ⇧ is an arbitrary set of policies with no assumed relationship with the reward function r ( c, a ) ; and the realizable setting, where the policy class and the reward function follow a linear structure, known as the linear contextual bandit problem.

Original: In both cases,we are interested in instance-dependent sample complexity bounds.
Revised: In both cases, we are interested in instance-dependent sample complexity bounds.

Original: That is, the upper and lowerbounds we seek do not simply depend on coarse quantities like | ⇧ | , |A| , and 1 / ✏ 2 , but more ﬁne-30 grained relationships between the context distribution ⌫ , geometry of policies ⇧ , and the rewardfunction r : C ⇥ A !
Revised: That is, the upper and lower bounds we seek do not simply depend on coarse quantities like | ⇧ | , |A| , and 1 / ✏ 2 , but more ﬁne- grained relationships between the context distribution ⌫ , geometry of policies ⇧ , and the reward function r : C ⇥ A !

Original: R .
Revised: R .

Original: Our motivation is that instance-dependent bounds describe the difﬁcultyof a particular problem instance, allowing optimal algorithms to adapt to the true difﬁculty of the problem, whether easy or hard.
Revised: Our motivation is that instance-dependent bounds describe the difﬁculty of a particular problem instance, allowing optimal algorithms to adapt to the true difﬁculty of the problem, whether easy or hard.

Original: We seek algorithms that take advantage of “easy” instances instead ofoptimizing for the worst-case [20].
Revised: We seek algorithms that take advantage of “easy” instances instead of optimizing for the worst-case [23].

Original: 1.1 Related work
Revised: 1.1 Related work

Original: Minimax regret bounds for general policy classes The vast majority of research in contextualbandits focuses on regret minimization.
Revised: Minimax regret bounds for general policy classes The vast majority of research in contextual bandits focuses on regret minimization.

Original: That is, for a time horizon T , the goal of the player isto minimize E
Revised: That is, for a time horizon T , the goal of the player is to minimize E

Original: [Equation]
Revised: [Equation]

Original: i .
Revised: i .

Original: The landmark algorithm EXP4 for non-stochasticmulti-armed bandits [4] achieves a regret bound of p |A| T log( | ⇧ | ) .
Revised: The landmark algorithm EXP4 for non-stochastic multi-armed bandits [5] achieves a regret bound of p |A| T log( | ⇧ | ) .

Original: Unfortunately, the running time40 of EXP4 is linear in | ⇧ | which is prohibitive for many problems of interest.
Revised: Unfortunately, the running time of EXP4 is linear in | ⇧ | which is prohibitive for many problems of interest.

Original: The algorithms proposedin [3, 9] achieve the same regret bound with a computational complexity that is only polynomialin T and log( | ⇧ | ) .
Revised: The algorithms proposed in [3, 11] achieve the same regret bound with a computational complexity that is only polynomial in T and log( | ⇧ | ) .

Original: Both approaches can be used to obtain an ✏ -optimal policy with probability atleast 1   using a sample complexity no more than |A| log( | ⇧ | /  ) ✏.
Revised: Both approaches can be used to obtain an ✏ -optimal policy with probability at least 1   using a sample complexity no more than |A| log( | ⇧ | /  ) ✏.

Original: None of these works made anyassumption on the connection between the reward function r and the policy class ⇧ .
Revised: None of these works made any assumption on the connection between the reward function r and the policy class ⇧ (i.e. the agnostic setting).

Original: Instance-dependent regret bounds for general policy classes
Revised: Instance-dependent regret bounds for general policy classes

Original: The epoch-greedy algorithmof [23] achieved the ﬁrst instance-dependent bounds on regret with a coarse guarantee depending only on the minimum policy gap  pol :
Revised: The epoch-greedy algorithm of [26] achieved the ﬁrst instance-dependent bounds on regret with a coarse guarantee depending only on the minimum policy gap  pol :

Original: = V ( ⇡ ⇤ )
Revised: = V ( ⇡ ⇤ )

Original:  max ⇡ 6 = ⇡ ⇤ V ( ⇡ ) .
Revised:  max ⇡ 6 = ⇡ ⇤ V ( ⇡ ) .

Original: In the pursuit of more48 ﬁne-grained regret bounds achievable by computationally efﬁcient algorithms, many authors resortto the realizability assumption [11–13, 31].
Revised: In the pursuit of more ﬁne-grained regret bounds achievable by computationally efﬁcient algorithms, many authors resort to the realizability assumption [14–16, 34].

Original: The learner knows a hypothesis class H where eachf 2 H is a map f : C ⇥ A !
Revised: The learner knows a hypothesis class H where each f 2 H is a map f : C ⇥ A !

Original: R , and there exists an f ⇤ 2 H such that r ( c, a ) =
Revised: R , and there exists an f ⇤ 2 H such that r ( c, a ) =

Original: f ⇤ ( c, a ) for all( c, a ) 2 C ⇥ A .
Revised: f ⇤ ( c, a ) for all ( c, a ) 2 C ⇥ A .

Original: Under this assumption, [13] proves lower and upper bounds on the instance-dependent52 regret.
Revised: Under this assumption, [16] proves lower and upper bounds on the instance-dependent regret.

Original: Their bounds are in term of the uniform gap  uniform :=
Revised: Their bounds are in term of the uniform gap  uniform :=

Original: min c
Revised: min c

Original: [Equation]
Revised: [Equation]

Original: A r ( c, ⇡ ⇤ ( c ))

Original:  r ( c, a ) .

Original: In general, for any policy class, they establish matching minimax lower and upper regret bounds ofthe form min {
Revised: In general, for any policy class, they establish matching minimax lower and upper regret bounds of the form min {

Original: [Equation]
Revised: [Equation]

Original:  H } , where C pol H is the policy disagreement coefﬁcient ,a parameter depending on the geometry of H and the context distribution ⌫ .
Revised: C [Equation] H } , where C pol H is the policy disagreement coefﬁcient , a parameter depending on the geometry of H and the context distribution ⌫ .

Original: That is, these boundshold with respect to a worst-case family of instances parameterized by  uniform and C pol H .
Revised: That is, these bounds hold with respect to a worst-case family of instances parameterized by  uniform and C pol H .

Original: Using thestandard online-to-batch conversion, this translates to a sample complexity (i.e. the time required toﬁnd an ✏ -good policy with constant probability) of roughly |A| log( |H| ) ✏  uniform C pol H .
Revised: Using the standard online-to-batch conversion, this translates to a sample complexity (i.e. the time required to ﬁnd an ✏ -good policy with constant probability) of roughly |A| log( |H| ) ✏  uniform C pol H .

Original: We show in Corollary A.1that this sample complexity is at least as large as our bounds.
Revised: We show in Corollary 2. that this sample complexity is at least as large as our bounds.

Original: Further, unlike our bounds below, thissample complexity is unbounded as ✏ goes to 0 .
Revised: Further, unlike our bounds below, this sample complexity is unbounded as ✏ goes to 0 .

Original: Recent work reﬁnes these kinds of regret boundsfurther, and provides minimax regret bounds in terms of the decision-estimation coefﬁcient [14].
Revised: Recent work reﬁnes these kinds of regret bounds further, and provides minimax regret bounds in terms of the decision-estimation coefﬁcient [17].

Original: Regret bounds for linear contextual bandits A special case of the realizable case assumes alinear structure for H .
Revised: Regret bounds for linear contextual bandits A special case of the realizable case assumes a linear structure for H .

Original: Assume there exists a known feature map  : C ⇥ A !
Revised: Assume there exists a known feature map  : C ⇥ A !

Original: R d and an unknown✓ ⇤ 2 R d such that the true reward function is given as r ( c, a ) =
Revised: R d and an unknown ✓ ⇤ 2 R d such that the true reward function is given as r ( c, a ) =

Original: h  ( c, a ) , ✓ ⇤ i .
Revised: h  ( c, a ) , ✓ ⇤ i .

Original: For this setting,popular optimism-based algorithms like LinUCB[24] and Thompson sampling[28, 30] achieve aregret bound of min { d p T, d 2  uniform } [1].
Revised: For this setting, popular optimism-based algorithms like LinUCB [27] and Thompson sampling [31, 33] achieve a regret bound of min { d p T, d 2  uniform } [1].

Original: Appealing to the online-to-batch conversion, this translatesto a PAC guarantee of d 2 ✏  uniform .
Revised: Appealing to the online-to-batch conversion, this translates to a PAC guarantee of d 2 ✏  uniform .

Original: More precise instance-dependent upper bounds on regret matchinstance-dependent lower bounds asymptotically as T ! 1 [16, 33].
Revised: More precise instance-dependent upper bounds on regret match instance-dependent lower bounds asymptotically as T ! 1 [19, 36].

Original: These works are most similarto our setting and have qualitatively similar style algorithms.
Revised: These works are most similar to our setting and have qualitatively similar style algorithms.

Original: However, both approaches rely onasymptotics with large problem-dependent terms that may dominate the bounds in ﬁnite time.
Revised: However, both approaches rely on asymptotics with large problem-dependent terms that may dominate the bounds in ﬁnite time.

Original: Ourwork is focused on upper bounds that nearly  lower bounds 
Revised: Our work is focused on upper bounds that nearly match lower bounds for all ﬁnite times.

Revised: Recently, instance-dependent sample complexity results for reinforcement learning in the tabular and linear function approximation settings have appeared [4, 39, 40].

Revised: As contextual bandits is a special case of ﬁnite-horizon reinforcement learning with a horizon length of 1 , their results immediately can be applied here.

Revised: However, the cost of this generality is that these algorithms have very large lower order terms (i.e., problem-dependent factors that multiply a 1 / ✏ term) making them far from optimal in our setting.

Revised: Moreover, the leading order term of [39] cannot be related to our lower bounds.

Original: PAC sample complexity for contextual bandits As we will describe, all contextual bandits withan arbitrary policy class can be reduced to PAC learning for linear bandits.
Revised: PAC sample complexity for contextual bandits As we will describe, all contextual bandits with an arbitrary policy class can be reduced to PAC learning for linear bandits.

Original: Once we made this reduction, our sample complexity analysis draws inspiration from the nearly instance-optimal algorithmfor linear best-arm identiﬁcation [10].
Revised: Once we made this reduction, our sample complexity analysis draws inspiration from the nearly instance-optimal algorithm for linear best-arm identiﬁcation [13].

Revised: The work in [10] provides a simple regret bound assuming a kernel structure on the reward function, while their bound is minimax and they assume a lower bound on eigenvalues of the covariance matrix of the context distribution.

Original: PAC sample complexity of linear contextual bandits was alsostudied in [35], who shows a minimax guarantee sample complexity that scales with d 2 ✏ 2 log(1 /  ) .
Revised: PAC sample complexity of linear contextual bandits was also studied in [41], who shows a minimax guarantee sample complexity that scales with d 2 ✏ 2 log(1 /  ) .

Original: Similar to our work, [3] deﬁne their action sampling distribution as a convex combination over policies.
Revised: Similar to our work, [3] deﬁne their action sampling distribution as a convex combination over policies.

Original: Our sampling distribution, as well as the optimal sampling distribution, cannot be representedthis way and is actually derived from the dual of the optimal experimental design objective.
Revised: Our sampling distribution, as well as the optimal sampling distribution, cannot be represented this way and is actually derived from the dual of the optimal experimental design objective.

Original: Contributions.
Revised: Contributions.

Original: In this work, our contributions include:
Revised: In this work, our contributions include:

Original: In the agnostic setting, we introduce a quantity ⇢ ⇧ that characterizes the instance-dependent sample complexity of PAC learning for contextual bandits (see Equation 1). 
Revised: In the agnostic setting, we introduce a quantity ⇢ ⇧ that characterizes the instance-dependent sample complexity of PAC learning for contextual bandits (see Equation 1). We show that

Revised: ⇢ ⇧ appears

Revised: in

Original: We show that ⇢ ⇧ appearsin information theoretic lower bound on the sample complexity[-of any PAC algorithm as ✏ ! 
-]
Revised: information theoretic lower bound on the sample complexity[-of any PAC algorithm as ✏ ! 
-]

Revised: of

Revised: any PAC algorithm as ✏ !

Original: in Theorem 2.2.
Revised: in Theorem 2.2.

Original: To ground this
Revised: To ground this,

Original: we describe
Revised: we describe

Original: it carefully 
Revised: it carefully in

Original: in the setting of the trivial policy class(Section 2.2) and
Revised: the setting of the trivial policy class ( Section

Original:  linear policy classes (Section 2.3).
Revised: and linear policy classes ( Section

Original: We construct an instance on which any regret minimax-optimal algorithm necessarily has a samplecomplexity that scales quadratically with the optimal sample complexity (Theorem 2.6).
Revised: We construct an instance on which any regret minimax-optimal algorithm necessarily has a sample complexity that scales quadratically with the optimal sample complexity (Theorem 2.6).

Original: This showsthat no algorithm can be both regret minimax-optimal and instance-optimal PAC.3.
Revised: This shows that no algorithm can be both regret minimax-optimal and instance-optimal PAC.

Original: Finally, we propose Algorithm 3 whose sample complexity nearly matches the lower bound basedon ⇢ ⇧ .
Revised: Finally, we propose Algorithm 3 whose sample complexity nearly matches the lower bound based on ⇢ ⇧ By appealing to an

Original: By appealing to an argmax oracle, this algorithm has a runtime polynomial in ⇢ ⇧ , 1 / ✏ ,log(1 /  ) , |A| , and log( | ⇧ | ) , assuming a unit cost of invoking the oracle.
Revised: argmax oracle this algorithm has a runtime polynomial in ⇢ ⇧ , 1 / ✏ , log(1 /  ) , |A| , and log( | ⇧ | ) , assuming a unit cost of invoking the oracle.

Original:  Problem statement and main results
Revised:  Problem statement and main results

Original: More formally, deﬁne F t =
Revised: More formally, deﬁne F t =

Original:  ( c 1 , a 1 , r 1 , . . .
Revised:  ( c 1 , a 1 , r 1 , . . .

Original: , c t , a t , r t ) as the natural  -algebra ﬁltration capturingall observed random variables up to time t .
Revised: , c t , a t , r t ) as the natural  -algebra ﬁltration capturing all observed random variables up to time t .

Revised: For simplicity, we assume Gaussian noise in some of our analysis.

Original: At each time t an algorithm deﬁnes a sampling rule F t 7!
Revised: At each time t an algorithm deﬁnes a sampling rule F t 7!

Original: A which deﬁnes a t +1 , an F t -measurable stopping time ⌧ 2 N , and a selection rule F t 7!
Revised: A which deﬁnes a t +1 , an {F t } t  1 -adapted stopping time ⌧ 2 N , and a selection rule F t 7!

Original: ⇧96 that is only called once at the stopping time t = ⌧ .
Revised: ⇧ that is only called once at the stopping time t = ⌧ .

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: Fix ✏  0 and  2 (0 , 1) .
Revised: Fix ✏  0 and  2 (0 , 1) .

Original: We say an algorithm is ( ✏ ,  ) -PAC for contextual banditswith policy class ⇧ , if for every instance, at the stopping time ⌧ 2 N with ⌧ < 1 almost surely, thealgorithm outputs
Revised: We say an algorithm is ( ✏ ,  ) -PAC for contextual bandits with policy class ⇧ , if for every instance, at the stopping time ⌧ 2 N with ⌧ < 1 almost surely, the algorithm outputs

Original: [Equation]
Revised: [Equation]

Original: The sample complexity of an ( ✏ ,  ) -PAC algorithm for contextual bandits is the time at which thealgorithm stops and outputs b ⇡ .
Revised: The sample complexity of an ( ✏ ,  ) -PAC algorithm for contextual bandits is the time at which the algorithm stops and outputs b ⇡ .

Original: The following quantity governs the sample complexity :
Revised: As we will discuss, the following quantity governs the sample complexity :

Original: [Equation]
Revised: [Equation]

Original: c, ⇡ ( c )) ]

Original: _ ✏ ).

Original: Here, for any countable set X we have that 4 X
Revised: Here, for any countable set X we have that 4 X

Original: = { p 2 R |X| :
Revised: = { p 2 R |X| :

Original: P x 2 X p x = 1 , p x  0 8 x 2 X}103 so that p c for every c 2 C deﬁnes a probability distribution over actions A .
Revised: P x [Equation] X p x = 1 , p x  0 8 x 2 X} so that p c for every c 2 C deﬁnes a probability distribution over actions A .

Original: In addition we use the104 notation a _ b := max { a, b } .
Revised: In addition we use the notation a [Equation]

Original: We begin with a necessary condition on the sample complexity for theparticular case of exact policy identiﬁcation ( ✏ = 0 ).
Revised: We begin with a necessary condition on the sample complexity for the particular case of exact policy identiﬁcation ( ✏ = 0 ).

Original: Theorem 2.2 (Lower bound) .
Revised: Theorem 2.2 (Lower bound) .

Original: Fix ✏ = 

Revised: Fix ✏ =

Original: and  2 (0 , 1) .
Revised: and  2 (0 , 1) .

Original: Moreover, ﬁx a contextual bandit instanceµ = ( ⌫ , r ) and a collection of policies ⇧ .
Revised: Moreover, ﬁx a contextual bandit instance µ = ( ⌫ , r ) and a collection of policies ⇧ .

Original: Then any (0 ,  ) -PAC algorithm for contextual banditssatisﬁes E µ [ ⌧ ]  ⇢ ⇧ , 0 log(1 / 2 . 4  ) .
Revised: Then any (0 ,  ) -PAC algorithm for contextual bandits satisﬁes E µ [ ⌧ ]  ⇢ ⇧ , 0 log(1 / 2 . 4  ) .

Original: The proof of the lower bound follows from standard information theoretic arguments [21].
Revised: The proof of the lower bound follows from standard information theoretic arguments [24].

Original: Thelower bound implicitly applies to learners that know the distribution ⌫ precisely.
Revised: The lower bound implicitly applies to learners that know the distribution ⌫ precisely.

Original: In practice, suchknowledge would never be available however the learner may have a large dataset of ofﬂine data.
Revised: In practice, such knowledge would never be available however the learner may have a large dataset of ofﬂine data.

Original: Assumption 1.
Revised: Assumption 1.

Original: Prior to starting the game, the learning algorithm is given a large dataset ofcontexts D = { c t } Tt =1 , where each c t is drawn IID from ⌫ for all t 2 [ T ] , and T =114 O ( poly (1 / ✏ , |A| , log(1 /  ) , log( | ⇧ | ))) .
Revised: Prior to starting the game, the learning algorithm is given a large dataset of contexts D = { c t } Tt =1 , where each c t is drawn IID from ⌫ for all t 2 [ T ] , and T = O ( poly (1 / ✏ , |A| , log(1 /  ) , log( | ⇧ | ))) .

Original: The above only assumes access to samples from the context distribution, not rewards or the valuefunction.
Revised: The above only assumes access to samples from the context distribution, not rewards or the value function.

Original: Importantly, since C could be uncountable, we do not assume D covers the support of ⌫ .
Revised: Importantly, since C could be uncountable, we do not assume D covers the support of ⌫ .

Original: Assumption 1 is satisﬁed, for example, in an e-commerce setting where the context is the demographicinformation about visitors to the site for which massive troves of historical data may be available.
Revised: Assumption 1 is satisﬁed, for example, in an e-commerce setting where the context is the demographic information about visitors to the site for which massive troves of historical data may be available.

Original: Other works in PAC learning have made similar assumptions [17].
Revised: Other works in PAC learning have made similar assumptions [20].

Original: We would like our algorithm to becomputationally efﬁcient in the sense that it makes a polynomial number of calls to what we refer toas argmax oracle.
Revised: We would like our algorithm to be computationally efﬁcient in the sense that it makes a polynomial number of calls to what we refer to as argmax oracle.

Original: Such an assumption is common in the contextual bandits literature [3, 9, 22].
Revised: Such an assumption is common in the contextual bandits literature [3, 11, 25].

Original: Deﬁnition 2.3 (Argmax oracle ( AMO )) .
Revised: Deﬁnition 2.3 (Argmax oracle ( AMO )) .

Original: The oracle AMO ( ⇧ , { ( c t , s t ) } nt =1 ) is an algorithm thatgiven contexts and cost vectors ( c 1 , s 1 ) , · · · , ( c n , s n ) 2 C ⇥ R |A| , returns arg max⇡
Revised: The oracle AMO ( ⇧ , { ( c t , s t ) } nt =1 ) is an algorithm that given contexts and cost vectors ( c 1 , s 1 ) , · · · , ( c n , s n ) 2 C ⇥ R |A| , returns arg max

Original: [Equation]
Revised: [Equation]

Original: P nt =1 s t ( ⇡ ( c t )) . The constrained argmax oracle C-AMO , given an upper bound l on the loss, returnsarg max ⇡ 
Revised: The constrained argmax oracle C-AMO , given an upper bound l on the loss, returns arg max ⇡ 2 ⇧ P nt

Original: [Equation]
Revised: [Equation]

Original:  subject to
Revised: ) subject to

Revised: P

Original: [Equation]
Revised: [Equation]

Original: In general we can implement AMO by calling to cost-sensitive classiﬁcation [5, 9] and C-AMOthrough a Lagrangian relaxation and a cost-sensitive classiﬁcation oracle [2, 7].
Revised: In general we can implement AMO by calling to cost-sensitive classiﬁcation [6, 11] and C-AMO through a Lagrangian relaxation and a cost-sensitive classiﬁcation oracle [2, 8].

Original: Our algorithm usesan argmax oracle as a subroutine at most polynomially in ✏  1 , log(1 /  ) , |A| and log( | ⇧ | ) .
Revised: Our algorithm uses an argmax oracle as a subroutine at most polynomially in ✏  1 , log(1 /  ) , |A| and log( | ⇧ | ) .

Original: In thissense, it is computationally efﬁcient.
Revised: In this sense, it is computationally efﬁcient.

Original: The following sufﬁciency result holds for general ✏  0 .
Revised: The following sufﬁciency result holds for general ✏  0 .

Original: Theorem 2.4 (Upper bound) .
Revised: Theorem 2.4 (Upper bound) .

Original: Fix ✏  0 and  2 (0 , 1) .
Revised: Fix ✏  0 and  2 (0 , 1) .

Original: Under Assumption 1, there exists a computationally efﬁcient ( ✏ ,  ) -PAC algorithm for contextual bandits that satisﬁes ⌧  ⇢ ⇧ , ✏ log( | ⇧ | log 2 (1 / ✏ ) /  ) log(1 /  ✏ ) , where  ✏ = max { ✏ , min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ )
Revised: Under Assumption 1, there exists a computationally efﬁcient ( ✏ ,  ) -PAC algorithm for contextual bandits that satisﬁes ⌧  ⇢ ⇧ , ✏ log( | ⇧ | log 2 (1 / ✏ ) /  ) log(1 /  ✏ ) , where  ✏ = max { ✏ , min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ )

Original:  V ( ⇡ ) } .
Revised:  V ( ⇡ ) } .

Original: Furthermore, this sample complexity never exceeds |A| (log( | ⇧ | )+log(1 /  ))log(1 / ✏ ) ✏.
Revised: Furthermore, this sample complexity never exceeds |A| (log( | ⇧ | )+log(1 /  ))log(1 / ✏ ) ✏.

Original: The second part of the theorem follows from the ﬁrst, since ⇢ ⇧ , ✏  2 |A| / ✏ 2 by taking p c,a = 1 / |A|135 for all ( c, a ) 2 C ⇥ A .
Revised: The second part of the theorem follows from the ﬁrst, since ⇢ ⇧ , ✏  2 |A| / ✏ 2 by taking p c,a = 1 / |A| for all ( c, a ) 2 C ⇥ A .

Original: 2.1 Inefﬁciency of low-regret algorithms
Revised: 2.1 Inefﬁciency of low-regret algorithms

Original: Computationally efﬁcient algorithms are known to exist, such as ILOVETOCONBANDITS [3],which achieve a minimax-optimal cumulative regret of p
Revised: Computationally efﬁcient algorithms are known to exist, such as ILOVETOCONBANDITS [3], which achieve a minimax-optimal cumulative regret of p

Original: T |A| log( | ⇧ | /  ) .
Revised: T |A| log( | ⇧ | /  ) .

Original: Inspecting the proofin [3], one can extract a sample complexity of ✏  2 |A| log( | ⇧ | /  ) from such results (which is also140 minimax optimal for PAC).
Revised: Inspecting the proof in [3], one can extract a sample complexity of ✏  2 |A| log( | ⇧ | /  ) from such results (which is also minimax optimal for PAC).

Original: The previous section showed that the sample complexity of our algorithm,
Revised: The previous section showed that the sample complexity of our algorithm,

Original: Theorem 2.4, nearly matches the instance-dependent lower bound of Theorem 2.2.
Revised: Theorem 2.4, nearly matches the instance-dependent lower bound of Theorem 2.2.

Original: In other words,our algorithm achieves a nearly optimal instance-dependent PAC sample complexity.
Revised: In other words, our algorithm achieves a nearly optimal instance-dependent PAC sample complexity.

Original: However, it isnatural to wonder if perhaps with a tighter analysis, the minimax regret optimal algorithm in [3] alsoobtains the instance-optimal PAC sample complexity.
Revised: However, it is natural to wonder if perhaps with a tighter analysis, the minimax regret optimal algorithm in [3] also obtains the instance-optimal PAC sample complexity.

Original: In this section, we show that this is not the case.
Revised: In this section, we show that this is not the case.

Original: Indeed, we show that any algorithm that is minimax regret optimal must have a sample complexitythat is at least quadratic in the optimal PAC sample complexity of some instance.
Revised: Indeed, we show that any algorithm that is minimax regret optimal must have a sample complexity that is at least quadratic in the optimal PAC sample complexity of some instance.

Original: Deﬁnition 2.5 (Hard instance) .
Revised: Deﬁnition 2.5 (Hard instance) .

Original: Fix m 2 N ,  2 (0 , 1] and let C =
Revised: Fix m 2 N ,  2 (0 , 1] and let C =

Original: [ m ] with uniform distribution,
Revised: [ m ] with uniform distribution,

Original: A = { 0 , 1 } .
Revised: A = { 0 , 1 } .

Original: For i = 1 , . . .
Revised: For i = 1 , . . .

Original: , m , let ⇡ i ( j )
Revised: , m , let ⇡ i ( j )

Original: = 1 { i = j } and deﬁne r ( i, j ) =
Revised: = 1 { i = j } and deﬁne r ( i, j ) =

Original:  1 { j = ⇡ 1 ( i ) } .
Revised:  1 { j = ⇡ 1 ( i ) } .

Original: Then149 V ( ⇡ 1 ) =  and V ( ⇡ i )
Revised: Then V ( ⇡ 1 ) =  and V ( ⇡ i )

Original: =
Revised: =

Original:  (1  2 /m ) for all i 2 C \ { 1 } .
Revised:  (1  2 /m ) for all i 2 C \ { 1 } .

Original: Note that for the hard instance, m = | ⇧ | .
Revised: Note that for the hard instance, m = | ⇧ | .

Original: If observations are corrupted by N (0 , 1) additive noise,then a straightforward calculation shows that ⇢ ? , 0( v ) = 4 /m (2  /m ) 

Revised: If observations are corrupted by N (0 , 1) additive noise, then a straightforward calculation shows that ⇢ ⇧ , 0 = 4 /m (2  /m )

Original: = m   2 for the hard instance.
Revised: = m   2 for the hard instance.

Original: Theorem 2.6.
Revised: Theorem 2.6.

Original: Fix  2 (0 , 1) and  2 (0 , 1] .
Revised: Fix  2 (0 , 1) and  2 (0 , 1] .

Original: We say an algorithm is an ↵ -minimax regret algorithmif for some ↵ > 0 and all T 2 N : 
Revised: We say an algorithm is an ↵ -minimax regret algorithm if for some ↵ > 0 and all T 2 N : [Equation] p ↵

Original: [Equation]
Revised: [Equation]

Original:  where the maximum is taken over all contextual bandit instances µ 0 = ( ⌫ 0 , r 0 ) and T c,a ( T ) =
Revised: A [Equation] T where the maximum is taken over all contextual bandit instances µ 0 = ( ⌫ 0 , r 0 ) and T c,a ( T ) =

Original: P
Revised: P

Original: Tt =1 1 { c t = c, a t = a } .
Revised: Tt =1 1 { c t = c, a t = a } .

Original: For any ↵ -minimax regret algorithm, it is (0 ,  ) -PAC if at a stoppingtime ⌧ it outputs the optimal policy ⇡
Revised: For any ↵ -minimax regret algorithm, it is (0 ,  ) -PAC if at a stopping time ⌧ it outputs the optimal policy ⇡

Original: ⇤ w. p. at least 1   .
Revised: ⇤ w. p. at least 1   .

Original: Any ↵ -minimax regret algorithm that is(0 ,  ) -PAC satisﬁes E µ [ ⌧ ]  m 2   2 log 2 (1 / 2 .
Revised: Any ↵ -minimax regret algorithm that is (0 ,  ) -PAC satisﬁes E µ [ ⌧ ]  m 2   2 log 2 (1 / 2 .

Original: 4  ) / 4 ↵ for the instance µ = ( ⌫ , r ) deﬁned in 2.5.
Revised: 4  ) / 4 ↵ for the instance µ = ( ⌫ , r ) deﬁned in 2.5.

Original: We point out that the minimax regret optimal rate takes ↵
Revised: We point out that the minimax regret optimal rate takes ↵

Original: = log( m ) =
Revised: = log( m ) =

Original: log( | ⇧ | ) .
Revised: log( | ⇧ | ) .

Original: Thus, taking  = 1and  = 0 . 1 , the minimax regret optimal algorithm has a PAC sample complexity of m 2 / log( m ) ;whereas the PAC sample complexity of our algorithm, Theorem 2.4, is just m log( m ) .
Revised: Thus, taking  =and  = 0 . 1 , the minimax regret optimal algorithm has a PAC sample complexity of m 2 / log( m ) ; whereas the PAC sample complexity of our algorithm, Theorem 2.4, is just m log( m ) .

Original: That is,algorithms with optimal minimax regret have a sample complexity that is at least nearly the optimal
Revised: That is, algorithms with optimal minimax regret have a sample complexity that is at least nearly the optimal

Original: PAC sample complexity squared .
Revised: PAC sample complexity squared .

Original: This demonstrates that no algorithm can simultaneously be minimaxregret optimal and obtain the optimal PAC sample complexity.
Revised: This demonstrates that no algorithm can simultaneously be minimax regret optimal and obtain the optimal PAC sample complexity.

Original: 2.2 Trivial policy class
Revised: 2.2 Trivial policy class

Original: As a warm-up to discussing linear policy classes, let us consider the simplest policy class.
Revised: As a warm-up to discussing linear policy classes, let us consider the simplest policy class.

Original: Deﬁnition 2.7 (Trivial policy class) .
Revised: Deﬁnition 2.7 (Trivial policy class) .

Original: Assume |C| < 1 and let ⇧ = { ⇡ ( c )
Revised: Assume |C| < 1 and let ⇧ = { ⇡ ( c )

Original: = a : ( c, a ) 2 C ⇥ A} sothat | ⇧ | = |A| |C| .
Revised: = a : ( c, a ) 2 C ⇥ A} so that | ⇧ | = |A| |C| .

Original: The trivial policy class has the ﬂexibility to predict any action a 2 A individually for each c 2 C .
Revised: The trivial policy class has the ﬂexibility to predict any action a 2 A individually for each c 2 C .

Original: Thisallows us to show that ⇢ ⇧ , 0( v )  max c 2 ⌫ c
Revised: This allows us to show that ⇢ ⇧ , 0  max c 2 ⌫ c

Original: P a 0   2 c,a 0 (see Appendix A.3).
Revised: P a 0   2 c,a 0 (see Appendix A.3).

Original: An immediate corollaryof Theorem 2.4 is obtained by simply noting that | ⇧ | = |A| |C| .
Revised: An immediate corollary of Theorem 2.4 is obtained by simply noting that | ⇧ | = |A| |C| .

Original: Corollary 2.8 (Trivial class, upper) .
Revised: Corollary 2.8 (Trivial class, upper) .

Original: Fix ✏ > 0 and  2 (0 , 1) .
Revised: Fix ✏ > 0 and  2 (0 , 1) .

Original: Let ⇧ be the trivial policy class appliedto some ﬁxed C , A spaces.
Revised: Let ⇧ be the trivial policy class applied to some ﬁxed C , A spaces.

Original: Then under Assumption 1 there exists a computationally efﬁcient ( ✏ ,  ) -PACalgorithm for contextual bandits that satisﬁes ⌧  min { A ✏  2 , max c 1 ⌫ c
Revised: Then under Assumption 1 there exists a computationally efﬁcient ( ✏ ,  ) -PAC algorithm for contextual bandits that satisﬁes ⌧  min { A ✏  2 , max c 1 ⌫ c

Original: P a 0   2 c,a 0 } ( |C| log( |A| ) +174 log(1 /  ))
Revised: P a 0   2 c,a 0 } ( |C| log( |A| ) + log(1 /  ))

Original: log(1 /  ✏ ) , where  ✏ = max { ✏ , min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ )
Revised: log(1 /  ✏ ) , where  ✏ = max { ✏ , min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ )

Original:  V ( ⇡ ) } .
Revised:  V ( ⇡ ) } .

Original: Furthermore, this samplecomplexity never exceeds |A| ( |C| log( |A| )
Revised: Furthermore, this sample complexity never exceeds |A| ( |C| log( |A| )

Original: +log(1 /  ))
Revised: +log(1 /  ))

Original: ✏log(1 / ✏ ) .
Revised: ✏log(1 / ✏ ) .

Original: Ignoring log factors, the minimax sample complexity of the trivial class is just ✏  2 |A| ( |C| +log(1 /  )) .
Revised: Ignoring log factors, the minimax sample complexity of the trivial class is just ✏  2 |A| ( |C| +log(1 /  )) .

Original: This is actually a somewhat surprising result, because it says lim  !
Revised: This is actually a somewhat surprising result, because it says lim  !

Original: E [ ⌧ ] log(1 /  ) !
Revised: E [ ⌧ ] log(1 /  ) !

Original: ✏  2 |A| whichis independent of |C| .
Revised: ✏  2 |A| which is independent of |C| .

Original: To see why this result is somewhat remarkable, if we played a best-armidentiﬁcation algorithm for each of the |C| contexts, then this would lead to a sample complexity of180 ✏  2 |C| · |A| log(1 /  ) .
Revised: To see why this result is somewhat remarkable, if we played a best-arm identiﬁcation algorithm for each of the |C| contexts, then this would lead to a sample complexity of ✏  2 |C| · |A| log(1 /  ) .

Original: It is somewhat of a surprise that such a natural strategy is not optimal.
Revised: It is somewhat of a surprise that such a natural strategy is not optimal.

Original: For181 intuition for why we can avoid the multiplicative |C| , note that to identify an ✏ -good policy among182 just two policies ( ⇡ , ⇡ ⇤ ) using uniform exploration requires just ✏  2 |A| log(1 /  ) samples.
Revised: For intuition for why we can avoid the multiplicative |C| , note that to identify an ✏ -good policy among just two policies ( ⇡ , ⇡ ⇤ ) using uniform exploration requires just ✏  2 |A| log(1 /  ) samples.

Original: When we183 have more than two policies, a union bound achieves the claimed result.
Revised: When we have more than two policies, a union bound achieves the claimed result.

Original: The minimax sample complexity of Corollary 2.8 (i.e., the second statement) is nearly tight:
Revised: The minimax sample complexity of Corollary 2.8 (i.e., the second statement) is nearly tight:

Original: Theorem 2.9 (Trivial class, lower) .
Revised: Theorem 2.9 (Trivial class, lower) .

Original: Fix ✏ > 0 and  2 (0 , 1 / 6) .
Revised: Fix ✏ > 0 and  2 (0 , 1 / 6) .

Original: Let ⇧ be the trivial policyclass applied to some ﬁxed C , A spaces.
Revised: Let ⇧ be the trivial policy class applied to some ﬁxed C , A spaces.

Original: Moreover, ﬁx a contextual bandit instance µ = ( ⌫ , r )and a collection of policies ⇧ .
Revised: Moreover, ﬁx a contextual bandit instance µ = ( ⌫ , r ) and a collection of policies ⇧ .

Original: Then any (0 ,  ) -PAC algorithm for contextual bandits satisﬁes
Revised: Then any (0 ,  ) -PAC algorithm for contextual bandits satisﬁes

Original: E µ [ ⌧ ]  max c 1 ⌫ c
Revised: E µ [ ⌧ ]  max c 1 ⌫ c

Original: P a   2 c,a log(1 / 2 . 4  ) .
Revised: P a   2 c,a log(1 / 2 . 4  ) .

Original: Furthermore, sup µ E µ [ ⌧ ]  ✏  2 |A| ( |C| + log(1 /  )) .
Revised: Furthermore, sup µ E µ [ ⌧ ]  ✏  2 |A| ( |C| + log(1 /  )) .

Original: 2.3 Linear policy class
Revised: 2.3 Linear policy class

Original: A particularly compelling model-class of policies is the set of linear policies.
Revised: A particularly compelling model-class of policies is the set of linear policies.

Original: Deﬁnition 2.10 (Linear policy class) .
Revised: Deﬁnition 2.10 (Linear policy class) .

Original: Fix a feature map  : C ⇥ A !
Revised: Fix a feature map  : C ⇥ A !

Original: R d and assume it is known tothe learner.
Revised: R d and assume it is known to the learner.

Original: Let ⇧ = { ⇡ ( c ) =
Revised: Let ⇧ = { ⇡ ( c ) =

Original: arg max a 2 A h  ( c, a ) , ✓ i , 8 ✓ 2 R d } .
Revised: arg max a 2 A h  ( c, a ) , ✓ i , 8 ✓ 2 R d } .

Original: We can consider two settings: the agnostic setting and the realizable setting.
Revised: We can consider two settings: the agnostic setting and the realizable setting.

Original: In the agnostic setting,there is no assumed relationship between the true reward function r ( c, a ) and  : C ⇥ A !
Revised: In the agnostic setting, there is no assumed relationship between the true reward function r ( c, a ) and  : C ⇥ A !

Original: R d .
Revised: R d .

Original: Inthis case, Theorem 2.4 applies directly by taking a cover of ⇧ .
Revised: In this case, Theorem 2.4 applies directly by taking a cover of ⇧ .

Original: Corollary 2.11 (Agnostic, upper bound) .
Revised: Corollary 2.11 (Agnostic, upper bound) .

Original: Fix ✏  0 and  2 (0 , 1) .
Revised: Fix ✏  0 and  2 (0 , 1) .

Original: Let ⇧ be the linear policy class in R d .
Revised: Let ⇧ be the linear policy class in R d .

Original: Under Assumption 1 there exists a computationally efﬁcient ( ✏ ,  ) -PAC algorithm for contextual bandits that satisﬁes ⌧  ⇢ ⇧ , ✏ · ( d log(1 / ✏ ) + log(1 /  )) log(1 /  ✏ ) where ✏ = max { ✏ , min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ )
Revised: Under Assumption 1 there exists a computationally efﬁcient ( ✏ ,  ) -PAC algorithm for contextual bandits that satisﬁes ⌧  ⇢ ⇧ , ✏ · ( d log(1 / ✏ ) + log(1 /  )) log(1 /  ✏ ) where  ✏ = max { ✏ , min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ )

Original:  V ( ⇡ ) } .
Revised:  V ( ⇡ ) } .

Original: Furthermore, this sample complexity never exceeds
Revised: Furthermore, this sample complexity never exceeds

Original: [Equation]
Revised: [Equation]

Original: Comparing to the lower bound of Theorem 2.2, the instance dependent upper bound of Corollary 2.11matches up to a factor of the dimension and negligible log factors.
Revised: Comparing to the lower bound of Theorem 2.2, the instance dependent upper bound of Corollary 2. matches up to a factor of the dimension and negligible log factors.

Original: In contrast to the “model-free” feelof the agnostic case, we can also consider a “model-based” type setting, i.e. the realizable setting.
Revised: In contrast to the “model-free” feel of the agnostic case, we can also consider a “model-based” type setting, i.e. the realizable setting.

Original: Deﬁnition 2.12 (Realizable) .
Revised: Deﬁnition 2.12 (Realizable) .

Original: We say the linear policy class is realizable if there exists a ✓ ⇤ 2 R dsuch that r ( c, a ) =
Revised: We say the linear policy class is realizable if there exists a ✓ ⇤ 2 R d such that r ( c, a ) =

Original: h  ( c, a ) , ✓ ⇤ i for all c 2 C and a 2 A .
Revised: h  ( c, a ) , ✓ ⇤ i for all c 2 C and a 2 A .

Original: Thus, for any ⇡ 2 ⇧ we have V ( ⇡ ) =
Revised: Thus, for any ⇡ 2 ⇧ we have V ( ⇡ ) =

Original: E c ⇠ ⌫ [ r ( c, ⇡ ( c ))]
Revised: E c ⇠ ⌫ [ r ( c, ⇡ ( c ))]

Original: =
Revised: =

Original: E c ⇠ ⌫ [ h  ( c, ⇡ ( c )) , ✓ ⇤ i ]
Revised: E c ⇠ ⌫ [ h  ( c, ⇡ ( c )) , ✓ ⇤ i ]

Original: = h  ⇡ , ✓ ⇤ i with  ⇡ :=
Revised: = h  ⇡ , ✓ ⇤ i with  ⇡ :=

Original: E c ⇠ ⌫ [  ( c, ⇡ ( c ))] .
Revised: E c ⇠ ⌫ [  ( c, ⇡ ( c ))] .

Original: Finally, at the207 start of the game the learner knows this model.
Revised: Finally, at the start of the game the learner knows this model.

Original: The setting in Deﬁnition 2.12 is commonly referred to as the linear contextual bandit problem[1].
Revised: The setting in Deﬁnition 2.12 is commonly referred to as the linear contextual bandit problem [1].

Original: Clearly, we have that ⇡ ⇤ ( c ) =
Revised: Clearly, we have that ⇡ ⇤ ( c ) =

Original: arg max a 2 A h  ( c, a ) ,
Revised: arg max a 2 A h  ( c, a ) ,

Original: ✓ ⇤ i .
Revised: ✓ ⇤ i .

Original: We begin by deﬁning a quantity210 fundamental to our sample complexity results:
Revised: We begin by deﬁning a quantity fundamental to our sample complexity results:

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.13 (Realizable, lower bound) .
Revised: Theorem 2.13 (Realizable, lower bound) .

Original: Fix ✏ = 

Revised: Fix ✏ =

Original: and  2 (0 , 1) .
Revised: and  2 (0 , 1) .

Original: Let ⇧ be the linear policy classin R d and assume it is realizable (see Deﬁnitions 2.10 and 2.12).
Revised: Let ⇧ be the linear policy class in R d and assume it is realizable (see Deﬁnitions 2.10 and 2.12).

Original: Any (0 ,  ) -PAC algorithm in thissetting satisﬁes E [ ⌧ ]  ⇢ lin , 0 · log(1 / 2 .
Revised: Any (0 ,  ) -PAC algorithm in this setting satisﬁes E [ ⌧ ]  ⇢ lin , 0 · log(1 / 2 .

Original: 4  ) .
Revised: 4  ) .

Original: We now state our nearly matching upper bound.
Revised: We now state our nearly matching upper bound.

Original: However, in this case we note that the algorithm isnot computationally efﬁcient.
Revised: However, in this case we note that the algorithm is not computationally efﬁcient.

Original: Theorem 2.14 (Realizable, upper bound) .
Revised: Theorem 2.14 (Realizable, upper bound) .

Original: Fix ✏  0 and  2 (0 , 1) .
Revised: Fix ✏  0 and  2 (0 , 1) .

Original: Let ⇧ be the linear policy classin R d and assume it is realizable (see Deﬁnitions 2.10 and 2.12).
Revised: Let ⇧ be the linear policy class in R d and assume it is realizable (see Deﬁnitions 2.10 and 2.12).

Original: Under Assumption 1 there exists an( ✏ ,  ) -PAC algorithm  for this setting that with probability at least 1   it satisﬁes
Revised: Under Assumption 1 there exists an ( ✏ ,  ) -PAC algorithm (see Algorithm 1) for this setting satisfying

Original: [Equation]
Revised: [Equation]

Original: where  ✏ = max { ✏ , min ⇡
Revised: where  ✏ = max { ✏ , min ⇡

Original: [Equation]
Revised: [Equation]

Original: Furthermore, this sample complexity never exceeds d ( d log(1 / ✏ )
Revised: Furthermore, this sample complexity never exceeds [Equation] log [Equation]

Original: [Table]
Revised: [Equation]

Original: We remark that the algorithm that achieves this upper bound is very different than popular optimismbased algorithms for linear contextual bandits e.g., UCB or Thompson sampling [1].
Revised: We remark that the algorithm that achieves this upper bound is very different than popular optimismbased algorithms for linear contextual bandits e.g., UCB or Thompson sampling [1].

Original: Indeed, ouralgorithm computes an experimental design and is related to instance-dependent linear bandit algorithms developed for best-arm identiﬁcation [8, 10, 32] and regret minimization [16, 33].
Revised: Indeed, our algorithm computes an experimental design and is related to instance-dependent linear bandit algorithms developed for best-arm identiﬁcation [9, 12, 13, 35] and regret minimization [19, 36].

Original: To ourknowledge, Theorem 2.14 provides the ﬁrst instance-dependent sample complexity for the PACsetting of linear contextual bandits.
Revised: To our knowledge, Theorem 2.14 provides the ﬁrst instance-dependent sample complexity for the PAC setting of linear contextual bandits.

Original: The most relevant work to Theorem 2.14 is the work of [35]which demonstrated a minimax sample complexity of d 2 / ✏ 2 log(1 /  ) .
Revised: The most relevant work to Theorem 2.14 is the work of [41] which demonstrated a minimax sample complexity of d 2 / ✏ 2 log(1 /  ) .

Original: Also, we remark that the lowerand upper bounds in this section require an additive Gaussian noise.
Revised: Also, we remark that the lower and upper bounds in this section require an additive Gaussian noise.

Original: Remark 2.15 (Agnostic vs. Realizable) .
Revised: Remark 2.15 (Agnostic vs. Realizable) .

Original: Contrasting the above results, we note that the samplecomplexity of the agnostic case is always bounded by |A| d/ ✏ 2 .
Revised: Contrasting the above results, we note that the sample complexity of the agnostic case is always bounded by |A| d/ ✏ 2 .

Original: whereas it never exceeds d 2 / ✏ 2 forthe realizable case.
Revised: whereas it never exceeds d 2 / ✏ 2 for the realizable case.

Original: This matches the intuition that when the number of actions is much larger thanthe dimension, assuming realizability can signiﬁcantly reduce the sample complexity.
Revised: This matches the intuition that when the number of actions is much larger than the dimension, assuming realizability can signiﬁcantly reduce the sample complexity.

Revised: 2.4 Comparison to the Disagreement Coefﬁcient

Revised: The work of [16] provides regret bounds in terms of instance-dependent quantities inspired by the disagreement coefﬁcient , a notion of complexity common in the active learning literature [18].

Original: Optimal Algorithms for Contextual Bandits3.1 Reduction to linear realizability and a simple elimination scheme
Revised: The following corollary relates our sample complexity to these notions of disagreement coefﬁcients. Deﬁne the policy disagreement coefﬁcient as [Equation] ✏ where ⇧ ✏ := { ⇡ 2 ⇧ : P ⌫ ( ⇡ ( c ) 6 = ⇡ ⇤ ( c ))  ✏ } and the cost-sensitive disagreement coefﬁcient as

Revised: [Equation]

Revised: ✏ .

Revised: The

Revised: AdaCB algorithm of [16] achieves a regret of roughly

Revised: R T =

Revised: O ⇣ min

Revised: [Equation]

Revised: Observe that at time T , given the outputs ⇡ 1 , ⇡ 2 , · · · , ⇡ T from AdaCB algorithm, one could return a (randomized) policy ˜ ⇡ which on observing a context, samples from the empirical distribution over the outputs.

Revised: By Markov’s inequality we have ˜ ⇡ , V ( ⇡ ⇤ )

Revised:  V (˜ ⇡ )

Revised:  O ( ✏ ) with constant probability for ✏ = R T T .

Revised: Therefore, an upper bound on the regret translates to a PAC sample complexity of |A| log( | ⇧ | ) ✏  uniform C pol ⇧ ( ✏ /  uniform ) or |A| log( | ⇧ | ) ✏

Revised: C csc ⇧ ( ✏ ) .

Revised: Finally, Corollary 2.16 shows that this sample complexity bound is at least as large as our upper bound, see Appendix A.5 for the proof.

Revised: Corollary 2.16.

Revised: Recall that  uniform := min c 2 C min a 2 A r ( c, ⇡ ⇤ ( c ))

Revised:  r ( c, a ) .

Revised: For any ✏ 0 > 0 we have that

Revised: [Equation]

Revised: Moreover, for all ✏ 0  0 we have that ⇢ ⇧ , ✏ 0 < 1 whenever  pol :

Revised: =

Revised: V ( ⇡ ⇤ )

Revised:  max ⇡ 6 = ⇡ ⇤ V ( ⇡ ) > 0 .

Revised:  Optimal Algorithms for Contextual Bandits 3.1 Reduction to linear realizability and a simple elimination scheme

Original: The astute reader may have noticed that if we ignore computation, Theorem 2.4 is actually animmediate corollary of Theorem 2.14 by taking  ( c, a )
Revised: The astute reader may have noticed that if we ignore computation, Theorem 2.4 is actually an immediate corollary of Theorem 2.14 by taking  ( c, a )

Original: = vec ( e c e > a ) 2 R |C|·|A| where e i is aone-hot encoded vector so that r ( c, a ) =
Revised: = vec ( e c e > a ) 2 R |C|·|A| where e i is a one-hot encoded vector so that r ( c, a ) =

Original: h  ( c, a ) , ✓ ⇤ i with ✓ ⇤ 2 R |C|·|A| .
Revised: h  ( c, a ) , ✓ ⇤ i with ✓ ⇤ 2 R |C|·|A| .

Original: This observation is key to238 our sample complexity results.
Revised: This observation is key to our sample complexity results.

Original: Recall  ⇡ :=
Revised: Recalling  ⇡ :=

Original: E c ⇠ ⌫ [  ( c, ⇡ ( c ))]  from Deﬁnition 2.12, we have that V ( ⇡ ) =
Revised: E c ⇠ ⌫ [  ( c, ⇡ ( c ))] ( from Deﬁnition 2.12), we have that V ( ⇡ ) =

Original: E  [ r ( c, ⇡ ( c ))]
Revised: E c ⇠ ⌫ [ r ( c, ⇡ ( c ))]

Original: =
Revised: =

Original: E  [ h  ( c, ⇡ ( c )) , ✓ ⇤ i ]
Revised: E c ⇠ ⌫ [ h  ( c, ⇡ ( c )) , ✓ ⇤ i ]

Original: =
Revised: =

Original: h  ⇡ , ✓ ⇤ i .
Revised: h  ⇡ , ✓ ⇤ i .

Original: We stress that C can be uncountable, andthus we would never actually instantiate any of these vectors.
Revised: We stress that C can be uncountable, and thus we would never actually instantiate any of the vectors  ( c, a ) .

Original: For notational convenience, deﬁne the feasible set of (context, action) probability distributions as⌦
Revised: For notational convenience, deﬁne the feasible set of (context, action) probability distributions as ⌦ = [Equation] a

Original: [Equation]
Revised: [Equation]

Original: o .
Revised: o .

Original: Note that for each context, p c := { w c,a / ⌫ c } a 2 A 2  Adeﬁnes a probability distribution over actions.
Revised: Note that for each context, p c := { w c,a / ⌫ c } a 2 A 2  A deﬁnes a probability distribution over actions.

Original: Also deﬁne A ( w ) :=
Revised: Also deﬁne A ( w ) :=

Original: P c,a w c,a  ( c, a )  ( c, a ) > forany w 2 ⌦ .
Revised: P c,a w c,a  ( c, a )  ( c, a ) > for any w 2 ⌦ .

Original: Under this notation, recalling the right hand side from Theorems 2.13 and 2.14 we have
Revised: Under this notation, recalling the right hand side from Theorems 2.13 and 2.14 we have

Original: [Equation]
Revised: [Equation]

Original: To show that the sample complexity of Theorem 2.4 is a corollary of Theorem 2.14, it sufﬁces toshow that equation (1) and the above display are equal.
Revised: To show that the sample complexity of Theorem 2.4 is a corollary of Theorem 2.14, it sufﬁces to show that equation (1) and the above display are equal.

Original: To see this, observe
Revised: To see this, observe

Original: [Equation]
Revised: [Equation]

Original: Due to this equivalence, the lower bound of Theorem 2.2 is also a corollary of Theorem 2.13.
Revised: Due to this equivalence, the lower bound of Theorem 2.2 is also a corollary of Theorem 2.13.

Original: Thelower bound of Theorem 2.13 follows almost immediately from the lower bound argument in [10].
Revised: The lower bound of Theorem 2.13 follows almost immediately from the lower bound argument in [13].

Original: The conclusion of this section is that from a sample complexity analysis alone, all that is left is toprove Theorem 2.14.
Revised: The conclusion of this section is that from a sample complexity analysis alone, all that is left is to prove Theorem 2.14.

Original: In the next section we propose an algorithm that achieves this sample complexitybut assumes precise knowledge of the context distribution ⌫ (this is relaxed in following sections).
Revised: In the next section we propose an algorithm that achieves this sample complexity but assumes precise knowledge of the context distribution ⌫ (this is relaxed in following sections).

Original: While the algorithm is highly impractical for a number of reasons, its analysis provides a great dealof intuition and motivation for our ﬁnal algorithm.
Revised: While the algorithm is highly impractical for a number of reasons, its analysis provides a great deal of intuition and motivation for our ﬁnal algorithm.

Original: 3.2 A simple, impractical, elimination-style algorithm
Revised: 3.2 A simple, impractical, elimination-style algorithm

Original: Algorithm 1 provides an initial elimination based method for the PAC-contextual bandit problem.
Revised: Algorithm 1 provides an initial elimination based method for the PAC-contextual bandit problem.

Original: The algorithm runs in stages.
Revised: The algorithm runs in stages.

Original: Before the start of each stage ` 2 N , the algorithm deﬁnes a distributionp ( ` ) c
Revised: Before the start of each stage ` 2 N , the algorithm deﬁnes a distribution p ( ` ) c

Original: 2 4 A for each c 2 C .
Revised: 2 4 A for each c 2 C .

Original: At each successive time t 2 [ n ` ] , it playsthe random action a t ⇠ p ( ` ) c tin response to context c t ⇠ ⌫ , and receives random reward r t with E [ r t | c t , a t ] =
Revised: At each successive time t 2 [ n ` ] , it plays random action a t ⇠ p ( ` ) c t in response to context c t ⇠ ⌫ , and receives random reward r t with E [ r t | c t , a t ] =

Original: h  ( c t , a t ) , ✓ ⇤ i .
Revised: h  ( c t , a t ) , ✓ ⇤ i .

Original: 259 Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: using the identity w ( ` ) c,a := ⌫ c p ( ` ) c,a .
Revised: using the identity w ( ` ) c,a := ⌫ c p ( ` ) c,a .

Original: Thus, if we set O t = A ( w ( ` ) )  1  ( c t , a t ) r t then E
Revised: Thus, if we set O t = A ( w ( ` ) )  1  ( c t , a t ) r t then E

Original: [ O t ] = ✓ ⇤ .
Revised: [ O t ] = ✓ ⇤ .

Original: Astraightforward calculation also shows that Cov ( O t ) =
Revised: A straightforward calculation also shows that Cov ( O t ) =

Original: A ( w ( ` ) )  1 if r t is perturbed with additiveunit variance noise.
Revised: A ( w ( ` ) )  1 if r t is perturbed with additive unit variance noise.

Original: Thus, an unbiased estimator of  ( ⇡ , ⇡ ⇤ ) :
Revised: Thus, an unbiased estimator of  ( ⇡ , ⇡ ⇤ ) :

Original: = V ( ⇡ ⇤ )  V ( ⇡ ) =
Revised: = V ( ⇡ ⇤ )  V ( ⇡ ) =

Original: h  ⇡ ⇤   ⇡ , ✓ ⇤ i263 is simply h  ⇡ ⇤   ⇡ , 1 n `
Revised: h  ⇡ ⇤   ⇡ , ✓ ⇤ i is simply h  ⇡ ⇤   ⇡ , 1 n `

Original: P t O t i which has variance 1 n ` k  ⇡ ⇤   ⇡ k 2 A ( w ( ` ) )  1 .
Revised: P [Equation] O t i which has variance 1 n ` k  ⇡ ⇤   ⇡ k 2 A ( w ( ` ) )  1 .

Original: Intuitively, h  ⇡ ⇤  ⇡ , 1 n`
Revised: Intuitively, h  ⇡ ⇤   ⇡ , 1 n

Original: [Equation]
Revised: [Equation]

Original: 1 so we can safely conclude that apolicy ⇡ is sub-optimal (i.e., ⇡ 6 = ⇡ ⇤ )
Revised: 1 so we can safely conclude that a policy ⇡ is sub-optimal (i.e., ⇡ 6 = ⇡ ⇤ )

Original: if there exists any policy ⇡ 0 such that h  ⇡0   ⇡ , 1 n `
Revised: if there exists any policy ⇡ 0 such that h  ⇡

Original: [Equation]
Revised: [Equation]

Revised:   ⇡ , 1 n

Original: [Equation]
Revised: [Equation]

Original:  This is the intuition behind Contextual RAGE (Algorithm 1), whichinherits its name from the best-arm identiﬁcation algorithm of [10] that inspired its strategy.
Revised: i [Equation] This is the intuition behind Contextual RAGE (Algorithm 1), which inherits its name from the best-arm identiﬁcation algorithm of [13] that inspired its strategy.

Original: However, while h  ⇡ ⇤   ⇡ , 1 n `
Revised: However, while h  ⇡ ⇤   ⇡ , 1 n `

Original: P t O t i is unbiased and has controlled variance, it is potentiallyheavy-tailed because w ( ` ) c,a can be arbitrarily small.
Revised: P t O t i is unbiased and has controlled variance, it is potentially heavy-tailed because w ( ` ) c,a can be arbitrarily small.

Original: Instead of trying to control w ( ` ) c,a and appealing to
Revised: Instead of trying to control w ( ` ) c,a and appealing to

Original: Bernstein’s inequality,  we use the robust mean estimator of Catoni [25].
Revised: Bernstein’s inequality, in line 7 we use the robust mean estimator of Catoni [28].

Original: We can then show:
Revised: We can then show:

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: ⇡
Revised: ⇡

Original: ⇤ 2 ⇧ ` and max ⇡ 2 ⇧ ` h  ⇡ ⇤   ⇡ , ✓ ⇤ i  4 ✏ ` for all ` > 1 w.p. at least 1   .
Revised: ⇤ 2 ⇧ ` and max ⇡ 2 ⇧ ` h  ⇡ ⇤   ⇡ , ✓ ⇤ i  4 ✏ ` for all ` > 1 w.p. at least 1   .

Original: The lemma states that if ⇧ ` is the active set of policies still under consideration, the optimal policy ⇡ ⇤is never discarded from ⇧ ` , and moreover, the quality of all policies remaining in ⇧ ` is getting betterand better.
Revised: The lemma states that if ⇧ ` is the active set of policies still under consideration, the optimal policy ⇡ ⇤ is never discarded from ⇧ ` , and moreover, the quality of all policies remaining in ⇧ ` is getting better and better.

Original: We are now ready to state the main sample complexity result, with proof in Appendix B.
Revised: We are now ready to state the main sample complexity result, with proof in Appendix B.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Fix any policy class ⇧ = { ⇡ : C !
Revised: Fix any policy class ⇧ = { ⇡ : C !

Original: A} ⇡ , distribution over contexts ⌫ ,  2 (0 , 1) ,✏  0 , and feature map  : C ⇥ A !
Revised: A} ⇡ , distribution over contexts ⌫ ,  2 (0 , 1) , ✏  0 , and feature map  : C ⇥ A !

Original: R d such that r ( c, a ) =
Revised: R d such that r ( c, a ) =

Original: h  ( c, a ) , ✓ ⇤ i
Revised: h  ( c, a ) , ✓ ⇤ i

Original: (this is withoutloss generality, as one can always take  ( c, a ) 
Revised: (w.l.o.g. one can always take  ( c, a ) =

Original: = vec ( e c e > a ) ).
Revised: vec ( e c e > a ) ).

Original: With probability at least 1   , if ⇡ = E c ⇠ ⌫ [  ( c, ⇡ ( c ))] and ⇡ ⇤ = arg max ⇡ h  ⇡ , ✓ ⇤ i then Contextual-RAGE returns a policy b ⇡ 2 ⇧279 such that V ( b ⇡ )  V ( ⇡ ⇤ )
Revised: With probability at least 1   , if  ⇡ = E c ⇠ ⌫ [  ( c, ⇡ ( c ))] and ⇡ ⇤ = arg max ⇡ h  ⇡ , ✓ ⇤ i then Contextual-RAGE returns a policy b ⇡ 2 ⇧ such that V ( b ⇡ )  V ( ⇡ ⇤ )

Original:  ✏ after taking at most
Revised:  ✏ after taking at most

Original: [Equation]
Revised: [Equation]

Original: samples, where c is an absolute constant and  = min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ 
Revised: samples, where c is an absolute constant and  = min ⇡ 2 ⇧ \ ⇡ ⇤ V ( ⇡ ⇤ )

Original: [Equation]

Original:  V ( ⇡ 
Revised:  V ( ⇡ ) .

Original: [Equation]

Original: 3.3 Towards a more efﬁcient algorithm
Revised: 3.3 Towards a more efﬁcient algorithm

Original: One major issue with Algorithm 1 is that it explicitly maintains a set of policies ⇧ ` from roundto round.
Revised: One major issue with Algorithm 1 is that it explicitly maintains a set of policies ⇧ ` from round to round.

Original: Since ⇧ could be exponential in |A| , this is a non-starter for any implementation.
Revised: Since ⇧ could be exponential in |A| , this is a non-starter for any implementation.

Original: As amotivation for our approach, we consider a non-elimination algorithm, Algorithm 2, as an intermediatestep.
Revised: As a motivation for our approach, we consider a non-elimination algorithm, Algorithm 2, as an intermediate step.

Original: It does not maintain ⇧ ` and instead just solves the optimization problem (2) over ⇧ .
Revised: It does not maintain ⇧ ` and instead just solves the optimization problem (2) over ⇧ .

Original: The designcomputed in (2) is chosen to ensure that for all ⇡ 2 ⇧ , | 
Revised: The design computed in (2) is chosen to ensure that for all ⇡ 2 ⇧ , | b  [Equation] b ⇡

Original: [Equation]
Revised: [Equation]

Revised:

Original:  ( ⇡ , ⇡ ⇤ ) |  2 ✏ `  1 + 14  ( ⇡ , ⇡ ⇤ )287 with high probability (Lemma C.3).
Revised: [Equation] )   ( ⇡ [Equation] ⇡ ⇤ ) |  2 ✏ `  1 + 14  ( ⇡ , ⇡ ⇤ ) with high probability (Lemma C.3).

Original: Equivalently, we estimate gaps up to a constant factor forpolicies with  ( ⇡ , ⇡ ⇤ ) >
Revised: Equivalently, we estimate gaps up to a constant factor for policies with  ( ⇡ , ⇡ ⇤ ) >

Original: ✏ ` , while our gap estimates are bounded by ✏ ` for those policies satisfying ( ⇡ , ⇡ ⇤ )  ✏ ` .
Revised: ✏ ` , while our gap estimates are bounded by ✏ ` for those policies satisfying  ( ⇡ , ⇡ ⇤ )  ✏ ` .

Original: This ensures that our choice of b ⇡ ` is good enough, i.e. satisﬁes V ( ⇡ ⇤ )
Revised: This ensures that our choice of b ⇡ ` is good enough, i.e. satisﬁes V ( ⇡ ⇤ )

Original:  V ( b ⇡ ` )  ✏ `290 with high probability.
Revised:  V ( b ⇡ ` )  ✏ ` with high probability.

Original: The full proof is in Appendix C.
Revised: The full proof is in Appendix C.

Original: Unfortunately, Algorithm 2 introduces additional problems.
Revised: Unfortunately, Algorithm 2 introduces additional problems.

Original: It is not clear whether solving (2) iscomputationally efﬁcient.
Revised: It is not clear whether solving (2) is computationally efﬁcient.

Original: Also, we need to ﬁnd an estimator b  l that is computationally efﬁcient evenif the policy space ⇧ is inﬁnite.
Revised: Also, we need to ﬁnd an estimator b  l that is computationally efﬁcient even if the policy space ⇧ is inﬁnite.

Original: In addition, it requires precise knowledge of ⌫ to even deﬁne thedomain of distributions ⌦ optimized over, and store the solution w 2 C ⇥ A explicitly.
Revised: In addition, it requires precise knowledge of ⌫ to even deﬁne the domain of distributions ⌦ optimized over, and store the solution w 2 C ⇥ A explicitly.

Original: But in general,such precise knowledge will not be available and is only estimable using past data (Assumption 1).
Revised: But in general, such precise knowledge will not be available and is only estimable using past data (Assumption 1).

Original: 3.4 An instance-optimal and computationally efﬁcient algorithm.
Revised: 3.4 An instance-optimal and computationally efﬁcient algorithm.

Original: In this section we provide Algorithm 3, which witnesses the guarantees of Theorem 2.4 for thegeneral agnostic contextual bandit problem.
Revised: In this section we provide Algorithm 3, which witnesses the guarantees of Theorem 2.4 for the general agnostic contextual bandit problem.

Original: We now address the caveats of the previous approaches.
Revised: We now address the caveats of the previous approaches.

Original: Access to Ofﬂine Data.
Revised: Access to Ofﬂine Data.

Original: By Assumption 1, we have access to a large amount of sampled ofﬂinecontexts D , where each c t 2 D is drawn IID from ⌫ .
Revised: By Assumption 1, we have access to a large amount of sampled ofﬂine contexts D , where each c t 2 D is drawn IID from ⌫ .

Original: Having access to D allows us to approximate
Revised: Having access to D allows us to approximate

Original: E c ⇠ ⌫ [ · ] with expectations over the empirical distribution E c ⇠ ⌫ D [ · ] , where ⌫ D is the uniform distribu-302 tion over historical data
Revised: E c ⇠ ⌫ [ · ] with expectations over the empirical distribution E c ⇠ ⌫ D [ · ] , where ⌫ D is the uniform distribu- tion over historical data

Original: D .
Revised: D .

Original: The number of ofﬂine contexts we need only scales logarithmically overthe size of the policy set ⇧ , more speciﬁcally, poly( |A| , ✏  1 , log( | ⇧ | ) , log(1 /  )) .
Revised: The number of ofﬂine contexts we need only scales logarithmically over the size of the policy set ⇧ , more speciﬁcally, poly( |A| , ✏  1 , log( | ⇧ | ) , log(1 /  )) .

Original: We quantify theprecise number of samples needed in Appendix D.2.
Revised: We quantify the precise number of samples needed in Appendix D.2.

Original: Input
Revised: Input

Original: [Equation]
Revised: [Equation]

Original: 1: Initialize
Revised: 1: Initialize

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 2: for ` = 1 , 2 , · · · , d log 2 (1 / ✏ ) e do 3: ✏
Revised: 2: for ` = 1 , 2 , · · · , d log 2 (1 / ✏ ) e do 3: ✏

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: ⇧
Revised: ⇧

Original: [Equation]
Revised: [Equation]

Original: Let n ` be the minimum value s.t.:
Revised: Let n ` be the minimum value s.t.:

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: with solution w (
Revised: with solution w (

Original: [Equation]
Revised: [Equation]

Original: For each t 2 [ n ` ] , get c t ⇠ ⌫ , pull a t ⇠ p ( ` )
Revised: For each t 2 [ n ` ] , get c t ⇠ ⌫ , pull a t ⇠ p ( ` )

Original: c t , observe reward r t 6:
Revised: c t , observe reward r t 6:

Original: Compute O t =
Revised: Compute O t =

Original: A ( w ( ` ) )
Revised: A ( w ( ` ) )

Original:  1  ( c t , a t ) r t .
Revised:  1  ( c t , a t ) r t .

Original: For ⇡ , ⇡ 0 2 ⇧ ` b
Revised: For ⇡ , ⇡ 0 2 ⇧ ` b

Original: [Equation]
Revised: [Equation]

Original: Update ⇧
Revised: Update ⇧

Original: [Equation]
Revised: [Equation]

Original: b 
Revised: b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , ⇡
Revised: , ⇡

Original: [Equation]
Revised: [Equation]

Original: > ✏
Revised: > ✏

Original: [Equation]
Revised: [Equation]

Original: 9: end for
Revised: 9: end for

Original: Output: ⇧ ` +
Revised: Output: ⇧ ` +

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: : Initialize:
Revised: : Initialize:

Original: [Table]
Revised: [Table]

Original: arbitrarily
Revised: arbitrarily

Original: [Table]
Revised: [Table]

Original: : for
Revised: : for

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: , · · · , d log 2 (
Revised: , · · · , d log 2 (

Original: [Table]
Revised: [Table]

Original: / ✏ ) e do
Revised: / ✏ ) e do

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original:  / (
Revised:  / (

Original: [Table]
Revised: [Table]

Original: 2 | ⇧ | ) 4:
Revised: 2 | ⇧ | ) 4:

Original: Let n ` be the minimum value s.t.:
Revised: Let n ` be the minimum value s.t.:

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original:  ✏
Revised:  ✏

Original: [Equation]
Revised: [Equation]

Original: (2) with solution w ( ` ) 5:
Revised: (2) with solution w ( ` ) 5:

Original: For each t 2 [ n ` ] , get c t ⇠ ⌫ , pull a t ⇠ p ( ` )
Revised: For each t 2 [ n ` ] , get c t ⇠ ⌫ , pull a t ⇠ p ( ` )

Original: c t , observe reward r t 6:
Revised: c t , observe reward r t 6:

Original: Compute O t =
Revised: Compute O t =

Original: A ( w ( ` ) )
Revised: A ( w ( ` ) )

Original:  1  ( c t , a t ) r t .
Revised:  1  ( c t , a t ) r t .

Original: For each ⇡ 2 ⇧ , let b  ` ( ⇡ , b ⇡ 
Revised: For each ⇡ 2 ⇧ , let b  ` ( ⇡ , b ⇡ `  1 )

Original: ` ) =
Revised: =

Original: Cat ( { h  ⇡   b ⇡ `  , O i i } n ` i =1 ) .
Revised: Cat ( { h  ⇡   b ⇡ `  1 , O i i } n ` i =1 ) .

Original: Set b ⇡ ` := arg min ⇡ 2 ⇧ b  ` ( ⇡ , b ⇡ `  ) (3) 9: end for
Revised: Set b ⇡ ` := arg min ⇡ 2 ⇧ b  ` ( ⇡ , b ⇡ `  1 ) (3) 9: end for

Original: Output: b ⇡ l
Revised: Output: b ⇡ l

Original: Computing the design efﬁciently.
Revised: Computing the design efﬁciently.

Original: As described, the context space C may be inﬁnite so maintaininga distribution !
Revised: As described, the context space C may be inﬁnite so maintaining a distribution !

Original: 2 ⌦ ⇢  C ⇥ A is not possible.
Revised: 2 ⌦ ⇢  C ⇥ A is not possible.

Original: To overcome this issue, we consider the dual problem308 of equation (2).
Revised: To overcome this issue, we consider the dual problem of equation (2).

Original: We can remove the square root by noticing that 2 p xy =
Revised: We can remove the square root by noticing that 2 p xy =

Original: min  > 0  x + y  , andintroducing an additional minimization over the variable  ⇡ , ⇡ 2 ⇧ .
Revised: min  > 0  x + y  , and introducing an additional minimization over the variable  ⇡ , ⇡ 2 ⇧ .

Original: Then, the dual problem becomes
Revised: Then, the dual problem becomes

Original: [Equation]
Revised: [Equation]

Original: Exchanging the order of the minimums on !
Revised: Exchanging the order of the minimums on !

Original: and  , somewhat surprisingly we have the close-formexpression (Lemma E.6)
Revised: and  , somewhat surprisingly we have the close-form expression (Lemma E.6)

Original: [Equation]
Revised: [Equation]

Original: where for ⇡ 0 2 ⇧ , t ( c ) a ( ⇡ 0 ) 2 { 0 , 1 } | ⇧ | with [ t ( c ) a ( ⇡ 0 )]
Revised: where for ⇡ 0 2 ⇧ , t ( c ) a ( ⇡ 0 ) 2 { 0 , 1 } | ⇧ | with [ t ( c ) a ( ⇡ 0 )]

Original: ⇡ := 1 { ⇡ ( c ) = a, ⇡ 0 ( c ) 6 = a } + 1 { ⇡ ( c ) 6 =a, ⇡ 0 ( c ) = a } and [    ] ⇡ :=
Revised: ⇡ := 1 { ⇡ ( c ) = a, ⇡ 0 ( c ) 6 = a } + 1 { ⇡ ( c ) 6 = a, ⇡ 0 ( c ) = a } and [    ] ⇡ :=

Original:  ⇡  ⇡ .
Revised:  ⇡  ⇡ .

Original: Interestingly, this value is achieved at a sampling distribution! , which is a non-linear function of  rather than a convex combination over policies (as in [3]).
Revised: Interestingly, this value is achieved at a sampling distribution ! , which is a non-linear function of  rather than a convex combination over policies (as in [3]).

Original: Because we have an expectation over contexts, this expectation can be replaced by an empiricalestimate using historical data, thus avoiding any issues with an inﬁnite context space.
Revised: Because we have an expectation over contexts, this expectation can be replaced by an empirical estimate using historical data, thus avoiding any issues with an inﬁnite context space.

Original: The ﬁnalalgorithm utilizing these observations found is in Algorithm 3.
Revised: The ﬁnal algorithm utilizing these observations found is in Algorithm 3.

Original: The main challenge is ﬁnding a solution to the design (5).
Revised: The main challenge is ﬁnding a solution to the design (5).

Original: First, we can reduce it to a saddle pointproblem over (  ,  ) by considering only a dyadic sequence of n 2 { 2 k : k 2 N } .
Revised: First, we can reduce it to a saddle point problem over (  ,  ) by considering only a dyadic sequence of n 2 { 2 k : k 2 N } .

Original: We use analternating ascent/descent method, with the caveat that  lives in a simplex, and  in a box.
Revised: We use an alternating ascent/descent method, with the caveat that  lives in a simplex, and  in a box.

Original: Bothspaces are deﬁned over a potentially inﬁnite set of policies ⇧ (in the worst case exponentially in |C| ).
Revised: Both spaces are deﬁned over a potentially inﬁnite set of policies ⇧ (in the worst case exponential in |C| ).

Original: To handle this, we use the Frank-Wolfe (FW) method on  .
Revised: To handle this, we use the Frank-Wolfe (FW) method on  .

Original: Referring to the iterates of FW as  t , FWguarantees that the size of the support of  t in each iterate grows by at most 1 .
Revised: Referring to the iterates of FW as  t , FW guarantees that the size of the support of  t in each iterate grows by at most 1 .

Original: Thus, if initialized asa 1 -sparse vector, we only need to maintain a sparse  t in each iteration.
Revised: Thus, if initialized as a 1 -sparse vector, we only need to maintain a sparse  t in each iteration.

Original: Each iterate of Frank-Wolfeinvolves computing
Revised: Each iterate of Frank-Wolfe involves computing

Original: [Equation]
Revised: [Equation]

Original: To do so, we show that we can appeal to a constrained argmax oracle ( AMO ) to run the Frank-Wolfealgorithm, a similar approach to [3].
Revised: To do so, we show that we can appeal to a constrained argmax oracle ( AMO ) to run the Frank-Wolfe algorithm, a similar approach to [3].

Original: For  we use a gradient descent procedure.
Revised: To optimize over  we use a gradient descent procedure.

Original: We will show that ineach iterate, the support of  is contained in that of  , and we can quantify the number of steps ofgradient descent needed to ﬁnd an ✏ -good solution.
Revised: We will show that in each iterate, the support of  is contained in that of  , and we can quantify the number of steps of gradient descent needed to ﬁnd an ✏ -good solution.

Original: Though h l (  ,  , n ) might not be convex in  , we nevertheless are able to argue that it has a unique minima and that gradient descent converges to thisminima.
Revised: Though h l (  ,  , n ) might not be convex in  , we nevertheless are able to argue that it has a unique minima and that gradient descent converges to this minima.

Original: We introduce our subroutine and further discuss the above claims in Appendix D.
Revised: We introduce our subroutine and further discuss the above claims in Appendix D.

Original: Regularized Estimator.
Revised: Regularized Estimator.

Original: While Algorithms 1 and 2 use a robust mean estimator as in equation (3),this estimator is impractical with a very large number of policies ⇧ .
Revised: While Algorithms 1 and 2 use a robust mean estimator as in equation (3), this estimator is impractical with a very large number of policies ⇧ .

Original: Instead, we use a regularized IPSestimator that can be computed using historical data and an argmax oracle.
Revised: Instead, we use a regularized IPS estimator that can be computed using historical data and an argmax oracle.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Let  l ,  l , n l = FW-GD ( ⇧ , |A| , b ⇡ l  1 , ✏ l ) .
Revised: Let  l ,  l , n l = FW-GD ( ⇧ , |A| , b ⇡ l  1 , ✏ l ) .

Original: These are the solutions to
Revised: These are the solutions to

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: For

Original: [Table]

Original: [Table]

Original: ` ]

Original: [Table] ⌫ [Table] i  p ( 
Revised: For i 2 [ n ` ] get c i ⇠ ⌫ , pull a i ⇠ p ( ` ) c i where p ( ` ) c s ,a s / q

Original: [Table]
Revised: [Equation]

Original: ) c i [Table] [Table]
Revised: ) , observe rewards r

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: b ⇡ l  1 ) , observe rewards r s 7:

Original: For each ⇡ 2 ⇧ , deﬁne the IPS estimator
Revised: For each ⇡ 2 ⇧ , deﬁne the IPS estimator

Original: [Equation]
Revised: [Equation]

Original: 8: set
Revised: 8: set

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) 9: end for
Revised: 9: end for

Original: Output: b ⇡ l
Revised: Output: b ⇡ l

Original: Algorithm 3 putsit all  together and Theorem 3.3 shows our main result.
Revised: Algorithm 3 puts all the pieces together and Theorem 3.3 shows our main result.

Original: Note that for exposition purposes, we have omitted some additional regularization terms in the optimization problemsthat have no effect on the sample complexity, but ensure ﬁnite-time convergence.
Revised: Note that for exposition purposes, we have omitted some additional regularization terms in the optimization problems that have no effect on the sample complexity, but ensure ﬁnite-time convergence.

Original: Appendix Eshows the full algorithm and the proof.
Revised: Appendix E shows the full algorithm and the proof.

Original: In what follows, poly 1 ( |A| , ✏  1 , log(1 /  )) · log( | ⇧ | ) and339 poly 2 ( |A| , ✏  1 , log(1 /  ) , log( | ⇧ | )) are polynomials in their arguments that speciﬁed in the appendix.
Revised: In what follows, poly 1 ( |A| , ✏  1 , log(1 /  )) · log( | ⇧ | ) and poly 2 ( |A| , ✏  1 , log(1 /  ) , log( | ⇧ | )) are polynomials in their arguments.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Fix set of policies ⇧ , context distribution ⌫ and reward function r ( c, a ) 2 [0 , 1] .
Revised: Fix set of policies ⇧ , context distribution ⌫ and reward function r ( c, a ) 2 [0 , 1] .

Original: Withprobability at least 1   , provided a history D whose size exceeds poly 1 ( |A| , ✏  1 , log(1 /  )) · log( | ⇧ | ) ,342 Algorithm 3 returns a policy b ⇡ satisfying V ( ⇡ ⇤ )
Revised: With probability at least 1   , provided a history D whose size exceeds poly 1 ( |A| , ✏  1 , log(1 /  )) · log( | ⇧ | ) , Algorithm 3 returns a policy b ⇡ satisfying V ( ⇡ ⇤ )

Original:  V ( b ⇡` )  ✏ in a number of samples not exceeding343 O ( ⇢ ⇤ , ✏ log( | ⇧ | log 2 (1 /  ✏ ) /  ) log 2 (1 /  ✏ ) where  ✏ := max { ✏ , min ⇡ 2 ⇧ V ( ⇡ ⇤ )
Revised:  V ( b ⇡ )  ✏ in a number of samples not exceeding O ( ⇢ ⇤ , ✏ log( | ⇧ | log 2 (1 /  ✏ ) /  ) log 2 (1 /  ✏ ) where  ✏ := max { ✏ , min ⇡ 2 ⇧ V ( ⇡ ⇤ )

Original:  V ( ⇡ ) } .
Revised:  V ( ⇡ ) } .

Original: In addition,
Revised: In addition,

Original: Algorithm is computationally efﬁcient and requires at mostpoly 2 ( |A| , ✏  1 , log(1 /  ) , log( | ⇧ | )) calls to a constrained argmax oracle.
Revised: Algorithm is computationally efﬁcient and requires at most poly 2 ( |A| , ✏  1 , log(1 /  ) , log( | ⇧ | )) calls to a constrained argmax oracle.

Original: Conclusion.
Revised: Conclusion.

Original: This work provides the ﬁrst instance-dependent lower bounds for the ( ✏ ,  ) -PACcontextual bandit problem.
Revised: This work provides the ﬁrst instance-dependent lower bounds for the ( ✏ ,  ) -PAC contextual bandit problem.

Original: One limitation of this work is that our analysis of Algorithm 3 does notimmediately extend to the realizable linear setting.
Revised: One limitation of this work is that our analysis of Algorithm 3 does not immediately extend to the realizable linear setting.

Original: That is, a computationally efﬁcient algorithmthat achieves the same bound is not known to exist.
Revised: That is, a computationally efﬁcient algorithm that achieves the same bound is not known to exist.

Original: In the general agnostic settings discussed inthis work, we proposed a computationally efﬁcient algorithm.
Revised: In the general agnostic settings discussed in this work, we proposed a computationally efﬁcient algorithm.

Original: A second limitation is the assumptionthat we have access to a large pool of ofﬂine data.
Revised: A second limitation is the assumption that we have access to a large pool of ofﬂine data.

Original: Because it seems necessary to plan with someinformation about the context distribution, it is not clear how one would completely remove suchan assumption and achieve the same sample complexity bounds.
Revised: Because it seems necessary to plan with some information about the context distribution, it is not clear how one would completely remove such an assumption and achieve the same sample complexity bounds.

Original: As with any recommender system,there is the potential for unintended consequences from optimizing just a single metric.
Revised: As with any recommender system, there is the potential for unintended consequences from optimizing just a single metric.

Original: Moreover,other potential pitfalls can arise, such as negative feedback loops, if our assumptions fail to hold inreal-world environments.
Revised: Moreover, other potential pitfalls can arise, such as negative feedback loops, if our assumptions fail to hold in real-world environments.

Original: Such consequences can be mitigated by tracking a diverse set of metrics.
Revised: Such consequences can be mitigated by tracking a diverse set of metrics.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section.•
Revised: [Yes] See Section.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Please see our conclusion.
Revised: [Yes] Please see our conclusion.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Please see our conclusion.
Revised: [Yes] Please see our conclusion.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?[N/A]
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original:  (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?[N/A]
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original:  (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]