Original: Contextual Bandits with Knapsacks for a Conversion Model
Revised: Contextual Bandits with Knapsacks for a Conversion Model

Original: Abstract
Revised: Abstract

Original: We consider contextual bandits with knapsacks, with an underlying structure between rewards generated and cost vectors suffered.
Revised: We consider contextual bandits with knapsacks, with an underlying structure between rewards generated and cost vectors suffered.

Original: We do so motivated by sales with commercial discounts.
Revised: We do so motivated by sales with commercial discounts.

Original: At each round, given the stochastic i.i.d.
Revised: At each round, given the stochastic i.i.d.

Original: context x t and the arm picked a t (corresponding, e.g., to a discount level), a customer conversion may be obtained, in which case a reward r ( a, x t ) is gained and vector costs c ( a t , x t ) are suffered (corresponding, e.g., to losses of earnings).
Revised: context x t and the arm picked a t (corresponding, e.g., to a discount level), a customer conversion may be obtained, in which case a reward r ( a, x t ) is gained and vector costs c ( a t , x t ) are suffered (corresponding, e.g., to losses of earnings).

Original: Otherwise, in the absence of a conversion, the reward and costs are null.
Revised: Otherwise, in the absence of a conversion, the reward and costs are null.

Original: The reward and costs achieved are thus coupled through the binary variable measuring conversion or the absence thereof.
Revised: The reward and costs achieved are thus coupled through the binary variable measuring conversion or the absence thereof.

Original: This underlying structure between rewards and costs is different from the linear structures considered by Agrawal and Devanur [2016]
Revised: This underlying structure between rewards and costs is different from the linear structures considered by Agrawal and Devanur [2016]

Original: (but we show that the techniques introduced in the present article may also be applied to the case of these linear structures).
Revised: (but we show that the techniques introduced in the present article may also be applied to the case of these linear structures).

Original: The adaptive policies exhibited solve at each round a linear program based on upper-confidence estimates of the probabilities of conversion given a and x .
Revised: The adaptive policies exhibited solve at each round a linear program based on upper-conﬁdence estimates of the probabilities of conversion given a and x .

Original: This kind of policy is most natural and achieves a regret bound of the typical order ( OPT /B ) √ T , where B is the total budget allowed, OPT is the optimal expected reward achievable by a static policy, and T is the number of rounds.
Revised: This kind of policy is most natural and achieves a regret bound of the typical order ( OPT /B ) √ T , where B is the total budget allowed, OPT is the optimal expected reward achievable by a static policy, and T is the number of rounds.

Original:  Introduction and Literature Review
Revised:  Introduction and Literature Review

Original: We consider the framework of stochastic multi-armed bandits, which has been extensively studied since the early works by Thompson [1933] and Robbins [1952].
Revised: We consider the framework of stochastic multi-armed bandits, which has been extensively studied since the early works by Thompson [1933] and Robbins [1952].

Original: Two recent (and complementary) surveys summarizing the latest research in the field were written by Lattimore and Szepesvári [2020] and Slivkins [2019].
Revised: Two recent (and complementary) surveys summarizing the latest research in the ﬁeld were written by Lattimore and Szepesvári [2020] and Slivkins [2019].

Original: On the one hand, we are particularly interested in the setting of contextual stochastic multi-armed bandits, preferably with some structural assumptions on the dependency between rewards and contexts: linear models (again, a rich literature, see, among many others,
Revised: On the one hand, we are particularly interested in the setting of contextual stochastic multi-armed bandits, preferably with some structural assumptions on the dependency between rewards and contexts: linear models (again, a rich literature, see, among many others,

Original: Chu et al.
Revised: Chu et al.

Original: [2011] and Abbasi-Yadkori et al.
Revised: [2011] and Abbasi-Yadkori et al.

Original: [2011], whose work marked a turning point), and, for [0 , 1] -valued rewards, logistic models (Filippi et al.
Revised: [2011], whose work marked a turning point), and, for [0 , 1] -valued rewards, logistic models (Filippi et al.

Original: [2010] and Faury et al.
Revised: [2010] and Faury et al.

Original: On the other hand, we are also particularly interested in stochastic multi-armed bandits with knapsacks , i.e., with cumulative vector-cost constraints to be abided by on top of maximizing the accumulated rewards.
Revised: On the other hand, we are also particularly interested in stochastic multi-armed bandits with knapsacks , i.e., with cumulative vector-cost constraints to be abided by on top of maximizing the accumulated rewards.

Original: The setting was introduced by Badanidiyuru et al.
Revised: The setting was introduced by Badanidiyuru et al.

Original: [2013, 2018] and a comprehensive summary of the results achieved since then may be found in Slivkins [2019, Chapter 10].
Revised: [2013, 2018] and a comprehensive summary of the results achieved since then may be found in Slivkins [2019, Chapter 10].

Original: The intersection of these two frameworks of interest is called contextual bandits with knapsacks [CBwK] and is the focus of the present article.
Revised: The intersection of these two frameworks of interest is called contextual bandits with knapsacks [CBwK] and is the focus of the present article.

Original: Literature review on CBwK.
Revised: Literature review on CBwK.

Original: The first approach to CBwK, by Badanidiyuru et al.
Revised: The ﬁrst approach to CBwK, by Badanidiyuru et al.

Original: [2014] and Agrawal et al.
Revised: [2014] and Agrawal et al.

Original: [2016], assumes a joint stochastic generation of triplets of contexts-rewardscosts, with no specific underlying structure, and makes the problem tractable by using as a benchmark a finite set of static policies.
Revised: [2016], assumes a joint stochastic generation of triplets of contexts-rewardscosts, with no speciﬁc underlying structure, and makes the problem tractable by using as a benchmark a ﬁnite set of static policies.

Original: As noted by Agrawal and Devanur [2016], picking this finite set may be uneasy, which is why they introduce instead a structural assumption of linear modeling: the (unknown) expected rewards and cost vectors depend linearly on the contexts.
Revised: As noted by Agrawal and Devanur [2016], picking this ﬁnite set may be uneasy, which is why they introduce instead a structural assumption of linear modeling: the (unknown) expected rewards and cost vectors depend linearly on the contexts.

Original: We consider a different modeling assumption, motivated by sales with commercial discounts (see
Revised: We consider a different modeling assumption, motivated by sales with commercial discounts (see

Original: Appendix A): general (known) reward and cost functions are considered but they are coupled via a 0/1–valued factor, called a (customer) conversion, obtained as the realization of a Bernoulli variable with parameter P ( a, x ) depending on the context x observed (customer’s characteristics) and the action a taken (discount level offered).
Revised: Appendix A): general (known) reward and cost functions are considered but they are coupled via a 0/1–valued factor, called a (customer) conversion, obtained as the realization of a Bernoulli variable with parameter P ( a, x ) depending on the context x observed (customer’s characteristics) and the action a taken (discount level offered).

Original: The probabilities P ( a, x ) are themselves modeled by a logistic regression, whose parameters may be learned through an adaptation of the techniques by Filippi et al.
Revised: The probabilities P ( a, x ) are themselves modeled by a logistic regression, whose parameters may be learned through an adaptation of the techniques by Filippi et al.

Original: [2010] and Faury et al.
Revised: [2010] and Faury et al.

Original: We do so in the first phase of the adaptive policy introduced in this article.
Revised: We do so in the ﬁrst phase of the adaptive policy introduced in this article.

Original: More details on the comparison of the new setting considered to known settings of CBwK may be found in Section 2.2.
Revised: More details on the comparison of the new setting considered to known settings of CBwK may be found in Section 2.2.

Original: Primal-dual approach.
Revised: Primal-dual approach.

Original: The second phase of the adaptive policy exhibited uses the primal-dual approach to a convex optimization problem—actually, a simple optimization problem given by a linear program.
Revised: The second phase of the adaptive policy exhibited uses the primal-dual approach to a convex optimization problem—actually, a simple optimization problem given by a linear program.

Original: This approach was already used in various ways for bandits with knapsacks, including
Revised: This approach was already used in various ways for bandits with knapsacks, including

Original: CBwK, to define policies based on the dual problem: this is explicit in the LagrangeBwK policy of Immorlica et al.
Revised: CBwK, to deﬁne policies based on the dual problem: this is explicit in the LagrangeBwK policy of Immorlica et al.

Original: [2019] and is implicit in the reward-minus-weighted-cost approach of Agrawal and Devanur [2016] and Agrawal et al.
Revised: [2019] and is implicit in the reward-minus-weighted-cost approach of Agrawal and Devanur [2016] and Agrawal et al.

Original: [2016], as we underline in the proof sketch of Sectionas well as in the discussion of Section 6.
Revised: [2016], as we underline in the proof sketch of Sectionas well as in the discussion of Section 6.

Original: However, we only use the primal-dual approach in the analysis and state our adaptive policy directly in terms of the primal problem, where we substituted upper-confidence estimates of the probabilities P ( a, x ) .
Revised: However, we only use the primal-dual approach in the analysis and state our adaptive policy directly in terms of the primal problem, where we substituted upper-conﬁdence estimates of the probabilities P ( a, x ) .

Original: We therefore end up with a most natural adaptive policy, which mimics the optimal static policy used as a benchmark.
Revised: We therefore end up with a most natural adaptive policy, which mimics the optimal static policy used as a benchmark.

Original: This direct primal statement of the policy actually also works for the setting of linear CBwK studied by Agrawal and
Revised: This direct primal statement of the policy actually also works for the setting of linear CBwK studied by Agrawal and

Original: Devanur [2016], as we show in Section 6.
Revised: Devanur [2016], as we show in Section 6.

Original: Policies based on such direct primal statements were already considered for bandits with knapsacks (see Li et al.
Revised: Policies based on such direct primal statements were already considered for bandits with knapsacks (see Li et al.

Original: [2021] and references therein)
Revised: [2021] and references therein)

Original: but do not seem easily extendable to CBwK.
Revised: but do not seem easily extendable to CBwK.

Original: Outline and main contributions.
Revised: Outline and main contributions.

Original: The first contribution of this article is a new structured setting of
Revised: The ﬁrst contribution of this article is a new structured setting of

Original: CBwK, based on a coupling between general rewards and cost vectors through conversions modeled based on a logistic regression; we present and discuss it in Section 2.1 (and explain its origins in
Revised: CBwK, based on a coupling between general rewards and cost vectors through conversions modeled based on a logistic regression; we present and discuss it in Section 2.1 (and explain its origins in

Original: Appendix A of the supplementary material).
Revised: Appendix A of the supplementary material).

Original: The adaptive policy introduced is described in Section 3.
Revised: The adaptive policy introduced is described in Section 3.

Original: Its first phase consists of learning the parameter of logistic regression and is adapted from Faury et al.
Revised: Its ﬁrst phase consists of learning the parameter of logistic regression and is adapted from Faury et al.

Original: Its second phase—and this is the second contribution of this article—directly solves a primal problem with optimistic conversion probabilities.
Revised: Its second phase—and this is the second contribution of this article—directly solves a primal problem with optimistic conversion probabilities.

Original: The analysis, which we believe is concise, elegant, and natural, is provided in Sections 4 (when the context distribution ν is known) and 5 (when ν is unknown).
Revised: The analysis, which we believe is concise, elegant, and natural, is provided in Sections 4 (when the context distribution ν is known) and 5 (when ν is unknown).

Original: As mentioned above, Section 6 draws the consequences of our second contribution for linear CBwK.
Revised: As mentioned above, Section 6 draws the consequences of our second contribution for linear CBwK.

Original: Notation.
Revised: Notation.

Original: Throughout the article, vectors are denoted with bold symbols.
Revised: Throughout the article, vectors are denoted with bold symbols.

Original: In particular, 0 anddenote the vectors with all components equal to 0 and 1 , respectively.
Revised: In particular, 0 anddenote the vectors with all components equal to 0 and 1 , respectively.

Original: With no additional subscript, ∥ v ∥ denotes the Euclidean norm of a vector v , while a subscript given by a non-negative symmetric matrix M refers to ∥ v ∥ M = √ v T M v .
Revised: With no additional subscript, (cid:107) v (cid:107) denotes the Euclidean norm of a vector v , while a subscript given by a non-negative symmetric matrix M refers to (cid:107) v (cid:107) M = √ v T M v .

Original:  Learning Protocol and Motivation
Revised:  Learning Protocol and Motivation

Original: We describe the learning protocol and objectives considered (Section 2.1) and explain why it is not covered by earlier works (Section 2.2).
Revised: We describe the learning protocol and objectives considered (Section 2.1) and explain why it is not covered by earlier works (Section 2.2).

Original: We also detail (Appendix A in the supplementary material) how this learning protocol was defined based on an industrial motivation in the banking sector: market share expansion for loans by granting discounts, under commercial budget constraints.
Revised: We also detail (Appendix A in the supplementary material) how this learning protocol was deﬁned based on an industrial motivation in the banking sector: market share expansion for loans by granting discounts, under commercial budget constraints.

Original: 2.1 Learning Protocol and Modeling Assumptions
Revised: 1 Learning Protocol and Modeling Assumptions

Original: We consider a finite action set A , including a special action a null called no-op, and a finite context set X ⊆ R n .
Revised: We consider a ﬁnite action set A , including a special action a null called no-op, and a ﬁnite context set X ⊆ R n .

Original: (We discuss and mitigate finiteness of X in Section 2.2.)
Revised: (We discuss and mitigate ﬁniteness of X in Section 2.2.)

Original: A scalar reward function r : A × X → [0 , 1] and a vector-valued cost function c : A × X → [0 , 1] d evaluate the performance of actions given the contexts.
Revised: A scalar reward function r : A × X → [0 , 1] and a vector-valued cost function c : A × X → [0 , 1] d evaluate the performance of actions given the contexts.

Original: There are several sources of costs to control: each corresponds to a component of c .
Revised: There are several sources of costs to control: each corresponds to a component of c .

Original: We assume that these function are known, and (with no loss of generality) that their ranges are [0 , 1] and [0 , 1] d .
Revised: We assume that these function are known, and (with no loss of generality) that their ranges are [0 , 1] and [0 , 1] d .

Original: The no-op action induces null reward and costs: r ( a null , x ) = 0 and c ( a null , x ) = 0 for all x ∈ X .
Revised: The no-op action induces null reward and costs: r ( a null , x ) = 0 and c ( a null , x ) = 0 for all x ∈ X .

Original: Contexts—which correspond, for instance, to customers’ characteristics, see Appendix A—are drawn sequentially according to some distribution ν , which may be known or unknown (we will deal with both cases).
Revised: Contexts—which correspond, for instance, to customers’ characteristics, see Appendix A—are drawn sequentially according to some distribution ν , which may be known or unknown (we will deal with both cases).

Original: At each round t ⩾ 1 , upon observing the context x t ∈ X drawn, the learner picks an action a t ∈ A , which corresponds, for instance, to an offer made to the customer t .
Revised: At each round t (cid:62) 1 , upon observing the context x t ∈ X drawn, the learner picks an action a t ∈ A , which corresponds, for instance, to an offer made to the customer t .

Original: If the latter accepts the offer, an event which we denote y t
Revised: If the latter accepts the offer, an event which we denote y t

Original: = 1 , then the learner obtains a reward r ( a t , x t ) and suffers some costs c ( a t , x t ) .
Revised: = 1 , then the learner obtains a reward r ( a t , x t ) and suffers some costs c ( a t , x t ) .

Original: When the customer declines the offer, we set y t = 0 , and null reward and costs are obtained.
Revised: When the customer declines the offer, we set y t = 0 , and null reward and costs are obtained.

Original: Thus, in both cases, the reward and costs may be written as r ( a t , x t ) y t and c ( a t , x t ) y t .
Revised: Thus, in both cases, the reward and costs may be written as r ( a t , x t ) y t and c ( a t , x t ) y t .

Original: We call y t the conversion and now explain how it is modeled.
Revised: We call y t the conversion and now explain how it is modeled.

Original: Modeling conversions.
Revised: Modeling conversions.

Original: We model each conversion y t as an independent Bernoulli random drawn, with parameter P ( a t , x t ) depending on the context x t and action a t ̸ = a null .
Revised: We model each conversion y t as an independent Bernoulli random drawn, with parameter P ( a t , x t ) depending on the context x t and action a t (cid:54) = a null .

Original: We further assume that these probabilities may be written as a logistic regression model, i.e., there exists a known transfer function φ : A \ { a null } × X → R m and some unknown parameter θ ⋆ ∈ R m such that ∀ x ∈ X , ∀ a ∈ A\{ a null } ,
Revised: We further assume that these probabilities may be written as a logistic regression model, i.e., there exists a known transfer function ϕ : A \ { a null } × X → R m and some unknown parameter θ (cid:63) ∈ R m such that ∀ x ∈ X , ∀ a ∈ A\{ a null } ,

Original: [Equation]
Revised: [Equation]

Original: where η ( x ) = 1 / (1+e − x) .
Revised: where η ( x ) = 1 / (1+ [Equation] − x

Original:  We assume that φ is normalized in a way that its Euclidean norm satisfies ∥ φ ∥ ⩽ 1 and that a bounded convex set Θ containing θ ⋆ is known.
Revised: [Equation] [Equation] We assume that ϕ is normalized in a way that its Euclidean norm satisﬁes (cid:107) ϕ (cid:107) (cid:54) 1 and that a bounded convex set Θ containing θ (cid:63) is known.

Original: Such a modeling is natural and opens the toolbox of logistic bandits; see Faury et al.
Revised: Such a modeling is natural and opens the toolbox of logistic bandits; see Faury et al.

Original: [2020] and references cited therein.
Revised: [2020] and references cited therein.

Original: We however note (and discuss this fact in Appendix C) that the logistic regression model above is slightly different from the one by Faury et al.
Revised: We however note (and discuss this fact in Appendix C) that the logistic regression model above is slightly different from the one by Faury et al.

Original: The concept of a conversion y for a round when the no-op action a null is played is void, and thus, we leave the probabilities P ( a null , x ) undefined, though by an abuse of notation, these quantities might appear but always multiplied by a 0 , given, e.g., by indicator functions like 1 { a ̸ = a null } , null rewards r ( a null , x ) , or null costs c ( a null , x ) .
Revised: The concept of a conversion y for a round when the no-op action a null is played is void, and thus, we leave the probabilities P ( a null , x ) undeﬁned, though by an abuse of notation, these quantities might appear but always multiplied by a 0 , given, e.g., by indicator functions like 1 { a (cid:54) = a null } , null rewards r ( a null , x ) , or null costs c ( a null , x ) .

Original: Policies: static vs. adaptive.
Revised: Policies: static vs. adaptive.

Original: The learner is given a number of rounds T and a maximal budget B (the same for all cost components, with no loss of generality: up to some normalization).
Revised: The learner is given a number of rounds T and a maximal budget B (the same for all cost components, with no loss of generality: up to some normalization).

Original: A static policy is a function π : X → P ( A ) , where P ( A ) is the set of probability distributions over A .
Revised: A static policy is a function π : X → P ( A ) , where P ( A ) is the set of probability distributions over A .

Original: As is traditional in the literature of CBwK (we recall below why this is the case), we take as benchmark the static policy π ⋆ with largest expected cumulative rewards under the condition that its cumulative costs abide by the budget constraints in expectation.
Revised: As is traditional in the literature of CBwK (we recall below why this is the case), we take as benchmark the static policy π (cid:63) with largest expected cumulative rewards under the condition that its cumulative costs abide by the budget constraints in expectation.

Original: More formally, π ⋆ achieves the maximum defining
Revised: More formally, π (cid:63) achieves the maximum deﬁning

Original: [Equation]
Revised: [Equation]

Original: under
Revised: under

Original: [Equation]
Revised: [Equation]

Original: where E X ∼ ν denotes an expectation solely over random contexts X following distribution ν , where π a ( X ) denotes the probability mass put by π ( X ) on a ∈ A , and where ⩽ is understood componentwise.
Revised: where E X ∼ ν denotes an expectation solely over random contexts X following distribution ν , where π a ( X ) denotes the probability mass put by π ( X ) on a ∈ A , and where (cid:54) is understood componentwise.

Original: Of course, the sums in the two expectations above are taken indifferently over A or A \ { a null } .
Revised: Of course, the sums in the two expectations above are taken indifferently over A or A \ { a null } .

Original: The learner uses an adaptive policy, i.e., a sequence of measurable functions p t : H t − 1 × X → P ( A ) indexed by t ⩾ 1 , where H =
Revised: The learner uses an adaptive policy, i.e., a sequence of measurable functions p t : H t − 1 × X → P ( A ) indexed by t (cid:62) 1 , where H =

Original: X
Revised: X

Original: ×
Revised: ×

Original: A × { 0 , 1 } .
Revised: A × { 0 , 1 } .

Original: Indeed, the history available to the learner at the beginning of the round t ⩾ 2 is summarized by h t − 1 =
Revised: Indeed, the history available to the learner at the beginning of the round t (cid:62) 2 is summarized by h t − 1 =

Original: ( x s , a s , y s ) s ⩽ t − 1 , and we define h 0 as the empty vector.
Revised: ( x s , a s , y s ) s (cid:54) t − 1 , and we deﬁne h 0 as the empty vector.

Original: Such a policy draws the action a t for round t ⩾ 1 independently at random according to p t ( h t − 1 , x t ) .
Revised: Such a policy draws the action a t for round t (cid:62) 1 independently at random according to p t ( h t − 1 , x t ) .

Original: We impose hard budget constraints on adaptive policies: they must satisfy
Revised: We impose hard budget constraints on adaptive policies: they must satisfy

Original: [Equation]
Revised: [Equation]

Original: Such adaptive policies are called feasible in the literature.
Revised: Such adaptive policies are called feasible in the literature.

Original: To abide by these hard constraints, we may restrict our attention to adaptive policies that pick Dirac masses on a null whenever one component of the cumulative costs is larger than B − 1 .
Revised: To abide by these hard constraints, we may restrict our attention to adaptive policies that pick Dirac masses on a null whenever one component of the cumulative costs is larger than B − 1 .

Original: At the same time, an adaptive policy should maximize the cumulative rewards obtained or, equivalently, minimize its regret:
Revised: At the same time, an adaptive policy should maximize the cumulative rewards obtained or, equivalently, minimize its regret:

Original: [Equation]
Revised: [Equation]

Original: B OXA: C ONTEXTUAL 
Revised: B OX ONTEXTUAL BANDITS

Original: KNAPSACKS[CB W K] FOR A
Revised: KNAPSACKS

Original: MODEL
Revised: MODEL

Original:  Possibly unknown parameters: context distribution ν on X ; probability of conversion given action and context P : A \ { a null } × X → [0 , 1] , modeled as P ( a, x ) = η (cid:0) φ ( a, x ) T θ ⋆ (cid:1) for some known transfer function φ : A\{ a null }×X → R m , with ∥ φ ∥ ⩽ 1 , and some unknown parameter θ ⋆ ∈ R m , lying in a known bounded convex set Θ .
Revised: Known ﬁnite no ﬁnite Possibly unknown parameters: context distribution ν on X ; probability of conversion given action and context P : A \ { a null } × X → [0 , 1] , modeled as P ( a, x ) = η (cid:0) ϕ ( a, x ) T θ (cid:63) (cid:1) for some known transfer function ϕ : A\{ a null }×X → R m , with (cid:107) ϕ (cid:107) (cid:54) 1 , and some unknown parameter θ (cid:63) ∈ R m , lying in a known bounded convex set Θ .

Original: For rounds t = 1 , 2 , 3 , . . .
Revised: For rounds t = 1 , 2 , 3 , . . .

Original: , T :
Revised: , T :

Original: Context x t ∼ ν is drawn independently of the past; 2.
Revised: Context x t ∼ ν is drawn independently of the past; 2.

Original: Learner observes x t and picks an action a t ∈ A ; 3.
Revised: Learner observes x t and picks an action a t ∈ A ; 3.

Original: Conversion y t ∈ { 0 , 1 } is drawn according to Ber (cid:0) P ( a t , x t ) (cid:1) ; 4.
Revised: Conversion y t ∈ { 0 , 1 } is drawn according to Ber (cid:0) P ( a t , x t ) (cid:1) ; 4.

Original: Learner observes y t , gets reward r ( a t , x t ) y t , and suffers costs c ( a t , x t ) y t .Goals:
Revised: Learner observes y t , gets reward r ( a t , x t ) y t , and suffers costs c ( a t , x t ) y t .

Original: Maximize (cid:88) t ⩽ T r ( a t , x t ) y t while controlling (cid:88) t ⩽ T c ( a t , x t ) y t ⩽ B
Revised: while controlling (cid:88) t (cid:54) T c ( a t , x t ) y t (cid:54) B

Original: It may be proved (along the same lines as Agrawal and Devanur [2016, Appendix B] do for a different model) that the optimal static policy π ⋆ obtains, on average and in expectation, a cumulative reward at least as good as the best feasible adaptive policy.
Revised: It may be proved (along the same lines as Agrawal and Devanur [2016, Appendix B] do for a different model) that the optimal static policy π (cid:63) obtains, on average and in expectation, a cumulative reward at least as good as the best feasible adaptive policy.

Original: Summary.
Revised: Summary.

Original: A summary of the learning protocol and of the goals is provided in Box A.
Revised: A summary of the learning protocol and of the goals is provided in Box A.

Original: We note here that rewards gained and vector costs suffered at round t in the case y t
Revised: We note here that rewards gained and vector costs suffered at round t in the case y t

Original: = 1 of a conversion could be stochastic with expectations r ( a t , x t ) and c ( a t , x t ) : our analysis and the regret bounds would be unchanged, as long as the expectation functions r and c are known.
Revised: = 1 of a conversion could be stochastic with expectations r ( a t , x t ) and c ( a t , x t ) : our analysis and the regret bounds would be unchanged, as long as the expectation functions r and c are known.

Original: 2.2 Discussion and Comparison to Existing Learning Protocols
Revised: 2.2 Discussion and Comparison to Existing Learning Protocols

Original: The setting described above may be reduced to the general setting of CBwK, as introduced by
Revised: The setting described above may be reduced to the general setting of CBwK, as introduced by

Original: Badanidiyuru et al.
Revised: Badanidiyuru et al.

Original: [2014] and Agrawal et al.
Revised: [2014] and Agrawal et al.

Original: Indeed, introduce independent Bernoulli variables y t,a with parameters P ( a, x t ) , for all a ∈ A \ { a null } , and set y t,a null = 0 .
Revised: Indeed, introduce independent Bernoulli variables y t,a with parameters P ( a, x t ) , for all a ∈ A \ { a null } , and set y t,a null = 0 .

Original: The vectors (cid:16) x t , (cid:0) r t ( a ) (cid:1) a ∈A , (cid:0) c t ( a ) (cid:1) a ∈A (cid:17) , where r t ( a ) =
Revised: The vectors (cid:16) x t , (cid:0) r t ( a ) (cid:1) a ∈A , (cid:0) c t ( a ) (cid:1) a ∈A (cid:17) , where r t ( a ) =

Original: r ( a, x t ) y t,a and c t ( a ) =
Revised: r ( a, x t ) y t,a and c t ( a ) =

Original: c ( a, x t ) y t,a are i.i.d., and upon picking action a t ∈ A , the obtained and observed rewards and cost vectors equal r t ( a t ) and c t ( a t ) .
Revised: c ( a, x t ) y t,a are i.i.d., and upon picking action a t ∈ A , the obtained and observed rewards and cost vectors equal r t ( a t ) and c t ( a t ) .

Original: When X is discrete, we may consider the set Π of base policies that map X to { δ a : a ∈ A} , the set of Dirac masses at some a ∈ A .
Revised: When X is discrete, we may consider the set Π of base policies that map X to { δ a : a ∈ A} , the set of Dirac masses at some a ∈ A .

Original: The convex hull of Π is the set of all static policies X
Revised: The convex hull of Π is the set of all static policies X

Original: → P ( A ) , against which we would like our policy to compete; but the adaptive policies by
Revised: → P ( A ) , against which we would like our policy to compete; but the adaptive policies by

Original: Badanidiyuru et al.
Revised: Badanidiyuru et al.

Original: [2014] and Agrawal et al.
Revised: [2014] and Agrawal et al.

Original: [2016] only compete with respect to the best single element in Π , not the best convex combination of elements of Π .
Revised: [2016] only compete with respect to the best single element in Π , not the best convex combination of elements of Π .

Original: The setting of linear CBwK
Revised: The setting of linear CBwK

Original: (Agrawal and Devanur [2016]) provides a structural link between contexts and expected rewards and cost vectors, but in a linear way that is incomparable to the setting of
Revised: (Agrawal and Devanur [2016]) provides a structural link between contexts and expected rewards and cost vectors, but in a linear way that is incomparable to the setting of

Original: CBwK for a conversion model introduced above.
Revised: CBwK for a conversion model introduced above.

Original: More details are given in Section 6.
Revised: More details are given in Section 6.

Original: We also mention that linear and logistic structural links between contexts (prices) and rewards or costs were studied in a non-contextual setting (i.e., not in CBwK) by Miao et al.
Revised: We also mention that linear and logistic structural links between contexts (prices) and rewards or costs were studied in a non-contextual setting (i.e., not in CBwK) by Miao et al.

Original: Their strategy bears some resemblance to the one by Agrawal and Devanur [2016], in particular, both consider an online convex optimization strategy as a subroutine.
Revised: Their strategy bears some resemblance to the one by Agrawal and Devanur [2016], in particular, both consider an online convex optimization strategy as a subroutine.

Original: All mentioned references consider a no-op action a null .
Revised: All mentioned references consider a no-op action a null .

Original: (It could be replaced by the existence of a standard action a no-cost always achieving null costs and possibly some positive rewards.)
Revised: (It could be replaced by the existence of a standard action a no-cost always achieving null costs and possibly some positive rewards.)

Original: On the contrary, none of the mentioned references assumes that the context X set is finite.
Revised: On the contrary, none of the mentioned references assumes that the context X set is ﬁnite.

Original: This is a technical necessity for a part of the adaptive policy introduced; see the discussion of computational complexity at the end of Section 3.
Revised: This is a technical necessity for a part of the adaptive policy introduced; see the discussion of computational complexity at the end of Section 3.

Original: But somehow, considering a finite set Π of policies, as in
Revised: But somehow, considering a ﬁnite set Π of policies, as in

Original: Badanidiyuru et al.
Revised: Badanidiyuru et al.

Original: [2014] and Agrawal et al.
Revised: [2014] and Agrawal et al.

Original: [2016], is a counterpart to assuming finiteness of X .
Revised: [2016], is a counterpart to assuming ﬁniteness of X .

Original: Also, Appendix F actually mitigates this restriction that X is finite: learning the logistic parameter θ ⋆ may be achieved with continuous contexts (see Phase 1 in Section 3); only the subsequent optimization part (Phase 2 in Section 3) requires finiteness of X .
Revised: Also, Appendix F actually mitigates this restriction that X is ﬁnite: learning the logistic parameter θ (cid:63) may be achieved with continuous contexts (see Phase 1 in Section 3); only the subsequent optimization part (Phase 2 in Section 3) requires ﬁniteness of X .

Original: We may thus well discretize only X for this Phase 2, which is exactly what Appendix F performs.
Revised: We may thus well discretize only X for this Phase 2, which is exactly what Appendix F performs.

Original: This mitigation comes with possible theoretical guarantees as Sections 4 and 5 reveal that the errors ε t ( a, x ) for learning θ ⋆ and P , obtained as outcomes of the first step of the analyses, are carried over in the subsequent steps, where the optimization part is evaluated.
Revised: This mitigation comes with possible theoretical guarantees as Sections 4 and 5 reveal that the errors ε t ( a, x ) for learning θ (cid:63) and P , obtained as outcomes of the ﬁrst step of the analyses, are carried over in the subsequent steps, where the optimization part is evaluated.

Original:  Description of the Adaptive Policy Considered
Revised:  Description of the Adaptive Policy Considered

Original: At each stage t ⩾ 1 , the policy first updates an estimator (cid:98) θ t − 1 of θ ⋆ based on the history h t −available so far, based on an adaptation of the Logistic-UCB1 algorithm by Faury et al.
Revised: At each stage t (cid:62) 1 , the policy ﬁrst updates an estimator (cid:98) θ t − 1 of θ (cid:63) based on the history h t −available so far, based on an adaptation of the Logistic-UCB1 algorithm by Faury et al.

Original: [2020], and deduces estimators (cid:98) P t − 1 ( a, x ) and upper confidence bounds U t − 1 ( a, x ) of the probabilities P ( a, x ) .
Revised: [2020], and deduces estimators (cid:98) P t − 1 ( a, x ) and upper conﬁdence bounds U t − 1 ( a, x ) of the probabilities P ( a, x ) .

Original: The policy then solves the corresponding estimated version of the optimization problem (2).
Revised: The policy then solves the corresponding estimated version of the optimization problem (2).

Original: We now describe the corresponding two steps.
Revised: We now describe the corresponding two steps.

Original: In the description below, quantities that depend on information available at round t − 1 (respectively, t ) are indexed by t − 1 (respectively, t ).
Revised: In the description below, quantities that depend on information available at round t − 1 (respectively, t ) are indexed by t − 1 (respectively, t ).

Original: Phase 0:
Revised: Phase 0:

Original: In case the cost constraints are about to be violated.
Revised: In case the cost constraints are about to be violated.

Original: To make sure cost constraints are never violated, whenever at least one of the components of the current cumulative costs is larger than B − 1 and could possibly be larger than B at the end of round t , we play a null (and we actually do so for the rest of the rounds).
Revised: To make sure cost constraints are never violated, whenever at least one of the components of the current cumulative costs is larger than B − 1 and could possibly be larger than B at the end of round t , we play a null (and we actually do so for the rest of the rounds).

Original: This corresponds to defining p t ( h t − 1 , x ) =
Revised: This corresponds to deﬁning p t ( h t − 1 , x ) =

Original: δ a null for all x ∈ X , where δ a null denotes the Dirac mass on a null .
Revised: δ a null for all x ∈ X , where δ a null denotes the Dirac mass on a null .

Original: Otherwise, we proceed as described below in Phaseand Phase 2.
Revised: Otherwise, we proceed as described below in Phaseand Phase 2.

Original: Phase 1: Learning θ ⋆ via an adapted Logistic-UCB1.
Revised: Phase 1: Learning θ (cid:63) via an adapted Logistic-UCB1.

Original: This first phase depends on a regularization parameter λ > 0 and on upper-confidence bonuses ε t ( a, x ) > 0 , both to be specified by the analysis.
Revised: This ﬁrst phase depends on a regularization parameter λ > 0 and on upper-conﬁdence bonuses ε t ( a, x ) > 0 , both to be speciﬁed by the analysis.

Original: At rounds t ⩾ 2 , we first maximize a regularized log-likelihood of the history h t − 1 : ˜ θ t − 1 ∈ argmax θ ∈R m t − 1 (cid:88)
Revised: At rounds t (cid:62) 2 , we ﬁrst maximize a regularized log-likelihood of the history h t − 1 : ˜ θ t − 1 ∈ argmax θ ∈

Original: [Equation]
Revised: [Equation]

Original: In the expression above, we read that we only gather information about θ ⋆ at those rounds s when a s ̸ = a null .
Revised: In the expression above, we read that we only gather information about θ (cid:63) at those rounds s when a s [Equation] = a null .

Original: When ˜ θ t − 1 does not belong to Θ , an ad hoc projection step corrects for this, if needed:
Revised: When ˜ θ t − 1 does not belong to Θ , an ad hoc projection step corrects for this, if needed:

Original: [Equation]
Revised: [Equation]

Original: where Ψ 
Revised: where Ψ t − 1 ( θ ) = [Equation]

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: W

Original: [Equation]
Revised: [Equation]

Original: We recall that the function ˙
Revised: We recall that the function ˙

Original: η denotes the derivative of η , i.e., ˙ η
Revised: η denotes the derivative of η , i.e., ˙ η

Original: ( x ) =
Revised: ( x ) =

Original: e − x / (cid:0) 1 + e − x (cid:1) 2 .
Revised: e − x / (cid:0) 1 + e − x (cid:1) 2 .

Original: We have ˙ η = η
Revised: We have ˙ η = η

Original: (1 − η ) .
Revised: (1 − η ) .

Original: By plug-in, we finally define estimators and upper-confidence bounds of the probabilities P ( a, x ) for a ̸ = a null and all x ∈ X :
Revised: By plug-in, we ﬁnally deﬁne estimators and upper-conﬁdence bounds of the probabilities P ( a, x ) for a (cid:54) = a null and all x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: =
Revised: and

Original: η (cid:0) φ ( a, x ) T (cid:98) θ t −1 (cid:1) and
Revised: U t −

Original: U t − 1 ( a, x ) = min (cid:8) (cid:98) P t − 1 ( a, x ) + ε t − 1 ( a, x ) , 1 (cid:9) .
Revised: [Equation] a x ) = min (cid:8) (cid:98) P t − 1 ( a, x ) + ε t − 1 ( a, x ) , 1 (cid:9) .

Original: For a null , no estimators or upper-confidence bounds need to be defined, as the quantities P ( a null , x ) are actually undefined.
Revised: For a null , no estimators or upper-conﬁdence bounds need to be deﬁned, as the quantities P ( a null , x ) are actually undeﬁned.

Original: Phase 2: Sampling, via solving an optimization problem with expected constraints.
Revised: Phase 2: Sampling, via solving an optimization problem with expected constraints.

Original: This phase relies on a conservative-budget parameter denoted by B T , which is only slightly smaller than B and whose exact value is to be specified by the analysis.
Revised: This phase relies on a conservative-budget parameter denoted by B T , which is only slightly smaller than B and whose exact value is to be speciﬁed by the analysis.

Original: We start with the case of a known context distribution ν .
Revised: We start with the case of a known context distribution ν .

Original: At round t = 1 , we play an arbitrary action in A \ { a null } .
Revised: At round t = 1 , we play an arbitrary action in A \ { a null } .

Original: At rounds t ⩾ 2 , if at least one component of the cumulative vector costs suffered so far is larger than B − 1 , we pick a t = a null .
Revised: At rounds t (cid:62) 2 , if at least one component of the cumulative vector costs suffered so far is larger than B − 1 , we pick a t = a null .

Original: Otherwise, we pick for p t ( h t − 1 , · ) the solution of the optimization problem OPT ( ν, U t − 1 , B T ) defined 1 in (2), and draw a t according to p t ( h t − 1 , x t ) .
Revised: Otherwise, we pick for p t ( h t − 1 , · ) the solution of the optimization problem OPT ( ν, U t − 1 , B T ) deﬁned 1 in (2), and draw a t according to p t ( h t − 1 , x t ) .

Original: B OX B: L OGISTIC -UCB1 FOR
Revised: B OX B: L OGISTIC -UCB1 FOR

Original:  PROBLEMS
Revised: TO OPT PROBLEMS

Original: Parameters: regularization parameter λ > 0 ; conservative-budget parameter B T ; upperconfidence bonuses ε s ( a, x ) > 0 , for s ⩾ 1 and ( a, x ) ∈ (cid:0)
Revised: Parameters: regularization parameter λ > 0 ; conservative-budget parameter B T ; upperconﬁdence bonuses ε s ( a, x ) > 0 , for s (cid:62) 1 and ( a, x ) ∈ (cid:0)

Original: A \ { a null } (cid:1) × X .
Revised: A \ { a null } (cid:1) × X .

Original: Round t = 1 : play an arbitrary action a 1 ∈ A \ { a null }
Revised: Round t = 1 : play an arbitrary action a 1 ∈ A \ { a null }

Original: At rounds t ⩾ 2 :
Revised: At rounds t (cid:62) 2 :

Original: Phase 

Revised: Phase

Original: If (cid:88) s ⩽ t −c ( a s , x s ) y s ⩽ ( B − 1) 1 is violated, then p t ( h t − 1 , x ) =
Revised: If (cid:88) s (cid:54) t −c ( a s , x s ) y s (cid:54) ( B − 1) 1 is violated, then p t ( h t − 1 , x ) =

Original: δ a null for all x
Revised: δ a null for all x

Original: Phase 1 Otherwise, compute a maximum-likelihood estimator ˜ θ t − 1 of θ ⋆ according to (3), compute its projection (cid:98) θ t − 1 onto Θ according to (4), and define, for a ̸
Revised: Phase 1 Otherwise, compute a maximum-likelihood estimator ˜ θ t − 1 of θ (cid:63) according to (3), compute its projection (cid:98) θ t − 1 onto Θ according to (4), and deﬁne, for a (cid:54)

Original: = a null : (cid:98) P t − 1 ( a, x ) =
Revised: = a null : (cid:98) P t − 1 ( a, x ) =

Original: η (cid:0) φ ( a, x ) T (cid:98) θ t − 1 (cid:1) and U t − 1 ( a, x ) = min (cid:110) (cid:98) P t − 1 ( a, x )+ ε t − 1 ( a, x ) , 1 (cid:111)
Revised: η (cid:0) ϕ ( a, x ) T (cid:98) θ t − 1 (cid:1) and U t − 1 ( a, x ) = min (cid:110) (cid:98) P t − 1 ( a, x )+ ε t − 1 ( a, x ) , 1 (cid:111)

Original: Phase 2 Compute the solution p t ( h t − 1 , · ) of
Revised: Phase 2 Compute the solution p t ( h t − 1 , · ) of

Original: OPT (cid:0) ˜
Revised: OPT (cid:0) ˜

Original: ν, U t − 1 , B T (cid:1) =
Revised: ν, U t − 1 , B T (cid:1) =

Original: max π : X→P ( A )
Revised: max π : X→P ( A )

Original: T E
Revised: T E

Original: X ∼ ˜ ν
Revised: X ∼ ˜ ν

Original: (cid:34)(cid:88) a ∈A r ( a, X )
Revised: (cid:34)(cid:88) a ∈A r ( a, X )

Original: U t − 1 ( a, X ) π a ( X ) (cid:35) under
Revised: U t − 1 ( a, X ) π a ( X ) (cid:35) under

Original: T E X ∼ ˜ ν
Revised: T E X ∼ ˜ ν

Original: (cid:34)(cid:88) a ∈A c ( a, X )
Revised: (cid:34)(cid:88) a ∈A c ( a, X )

Original: U t − 1 ( a, X ) π a ( X ) (cid:35) ⩽ B T 1 , where ˜ ν denotes either ν
Revised: U t − 1 ( a, X ) π a ( X ) (cid:35) (cid:54) B T 1 , where ˜ ν denotes either ν

Original: (when it is known) or its empirical estimate (cid:98) ν t in (6) Draw an arm a t ∼ p t ( h t − 1 , x t ) .
Revised: (when it is known) or its empirical estimate (cid:98) ν t in (6) Draw an arm a t ∼ p t ( h t − 1 , x t ) .

Original: When the context distribution is unknown, we rather pick for p t ( h t − 1 , · ) the solution of the optimization problem OPT (cid:0)(cid:98) ν t , U t − 1 , B T (cid:1) , where
Revised: When the context distribution is unknown, we rather pick for p t ( h t − 1 , · ) the solution of the optimization problem OPT (cid:0)(cid:98) ν t , U t − 1 , B T (cid:1) , where

Original: [Equation]
Revised: [Equation]

Original: with δ x denoting the Dirac mass at x ∈ X .
Revised: with δ x denoting the Dirac mass at x ∈ X .

Original: Since x t is revealed at the beginning of round t , before we pick an action, we may indeed use (cid:98) ν t at round t .
Revised: Since x t is revealed at the beginning of round t , before we pick an action, we may indeed use (cid:98) ν t at round t .

Original: Summary and discussion of the computational complexity.
Revised: Summary and discussion of the computational complexity.

Original: We summarize the considered adaptive policy in Box B and now discuss its computational complexity.
Revised: We summarize the considered adaptive policy in Box B and now discuss its computational complexity.

Original: As ln φ and ln(1 − φ ) are strictly concave and smooth, the maximum-likelihood step (3) of Phaseconsists of maximizing a strictly concave and smooth function over R m , which may be performed efficiently.
Revised: As ln ϕ and ln(1 − ϕ ) are strictly concave and smooth, the maximum-likelihood step (3) of Phaseconsists of maximizing a strictly concave and smooth function over R m , which may be performed efﬁciently.

Original: The projection step (4) of Phase 1 is however an issue, both with the version of LogisticUCB1 discussed here and with the earlier approach by Filippi et al.
Revised: The projection step (4) of Phase 1 is however an issue, both with the version of LogisticUCB1 discussed here and with the earlier approach by Filippi et al.

Original: [2010, Section 3].
Revised: [2010, Section 3].

Original: The latter and
Revised: The latter and

Original: Faury et al.
Revised: Faury et al.

Original: [2020, Section 4.1] both underline that the projection step (4) is a complex optimization problem that however does not often need to be solved in practice, as they usually observe ˜ θ t − 1 ∈ Θ .
Revised: [2020, Section 4.1] both underline that the projection step (4) is a complex optimization problem that however does not often need to be solved in practice, as they usually observe ˜ θ t − 1 ∈ Θ .

Original: Our numerical experiments concur with this statement (but admittedly, they rely on choosing a rather large value of Θ ).
Revised: Our numerical experiments concur with this statement (but admittedly, they rely on choosing a rather large value of Θ ).

Original: On the contrary, Phase 2 of the adaptive policy consists of solving a linear program with |X| × |A| constraints, where where |X| and A denote the cardinality of X and A , respectively—see the detailed rewriting (13) in the supplementary material.
Revised: On the contrary, Phase 2 of the adaptive policy consists of solving a linear program with |X| × |A| constraints, where where |X| and A denote the cardinality of X and A , respectively—see the detailed rewriting (13) in the supplementary material.

Original: Therefore, the computational complexity of Phase 2 is polynomial (of weak order) in |X| × |A| .
Revised: Therefore, the computational complexity of Phase 2 is polynomial (of weak order) in |X| × |A| .

Original: To achieve this acceptable complexity we had however to restrict our attention to finite sets of contexts X , which requires in practice segmenting countable or continuous context sets into finitely many clusters, for instance.
Revised: To achieve this acceptable complexity we had however to restrict our attention to ﬁnite sets of contexts X , which requires in practice segmenting countable or continuous context sets into ﬁnitely many clusters, for instance.

Original: We do so in our experiments.
Revised: We do so in our experiments.

Original: Simulation study.
Revised: Simulation study.

Original: A simulation study on partially simulated but realistic data may be found in
Revised: A simulation study on partially simulated but realistic data may be found in

Original: Appendix F.
Revised: Appendix F.

Revised: The underlying dataset is the standard “default of credit card clients” dataset of UCI [2016], initially provided by Yeh and Lien [2009].

Revised: (It may be used under a Creative Commons

Revised: Attribution 4.0 International [CC BY 4.0] license.)

Original:  Analysis for a Known Context Distribution ν
Revised:  Analysis for a Known Context Distribution ν

Original: Since Θ is bounded, the following quantity, standardly introduced in the context of logistic bandits (see Faury et al.
Revised: Since Θ is bounded, the following quantity, standardly introduced in the context of logistic bandits (see Faury et al.

Original: [2020] and references therein), is finite, though possibly large:
Revised: [2020] and references therein), is ﬁnite, though possibly large:

Original: [Equation]
Revised: [Equation]

Original: We denote by ∥ Θ ∥ = max (cid:8) ∥ θ ∥ : θ ∈ Θ (cid:9) the maximal Euclidean norm of an element in Θ .
Revised: We denote by (cid:107) Θ (cid:107) = max (cid:8) (cid:107) θ (cid:107) : θ ∈ Θ (cid:9) the maximal Euclidean norm of an element in Θ .

Original: By construction, given that individual cost vectors lie in [0 , 1] d and due to its “Phase 0”, the adaptive policy considered always satisfies the budget constraints.
Revised: By construction, given that individual cost vectors lie in [0 , 1] d and due to its “Phase 0”, the adaptive policy considered always satisﬁes the budget constraints.

Original: The bound on rewards reads as follows.
Revised: The bound on rewards reads as follows.

Original: Theorem 1.
Revised: Theorem 1.

Original: In the setting of Box A of Section 2.1, we consider the adaptive policy of Box B of
Revised: In the setting of Box A of Section 2.1, we consider the adaptive policy of Box B of

Original: Section 3 assuming that the distribution of the contexts is known, i.e., with ˜ ν
Revised: Section 3 assuming that the distribution of the contexts is known, i.e., with ˜ ν

Original: = ν .
Revised: = ν .

Original: We set a confidence level 1 − δ ∈ (0 , 1) and use parameters λ
Revised: We set a conﬁdence level 1 − δ ∈ (0 , 1) and use parameters λ

Original: = m ln(1 + T/m ) ,
Revised: = m ln(1 + T/m ) ,

Original: [Equation]
Revised: [Equation]

Original: and ε t ( a, x ) stated in (9) of the supplementary material.
Revised: and ε t ( a, x ) stated in (9) of the supplementary material.

Original: Then, provided that T ⩾ 2 m and B > 4 + 2 (cid:112) 2 T ln(4 d/δ ) , we have, with probability at least 1 − 2 δ ,
Revised: Then, provided that T (cid:62) 2 m and B > 4 + 2 (cid:112) 2 T ln(4 d/δ ) , we have, with probability at least 1 − 2 δ ,

Original: OPT
Revised: OPT

Original: [Equation]
Revised: [Equation]

Original: , P,

Original: [Equation]

Original: −

Original: [Equation]

Original: where the closed-form expression of E T =
Revised: where the closed-form expression of E T =

Original: O (cid:0) m √ T ln T (cid:1) is in (35) of the supplementary material.
Revised: O (cid:0) m √ T ln T (cid:1) is in (35) of the supplementary material.

Original: We will rather discuss the bound of the more general Theorem 2 (to be stated and proved in Section 5) than the one of Theorem 1.
Revised: We will rather discuss the bound of the more general Theorem 2 (to be stated and proved in Section 5) than the one of Theorem 1.

Original: We provide a proof sketch in Section 4.1 and discuss the main technical novelty in Section 4.2.
Revised: We provide a proof sketch in Section 4.1 and discuss the main technical novelty in Section 4.2.

Original: 4.1 Proof Sketch for Theorem
Revised: 4.1 Proof Sketch for Theorem

Original: The detailed proof of Theorem 1 may be found in Appendix B.
Revised: The detailed proof of Theorem 1 may be found in Appendix B.

Original: We provide here an overview thereof, highlighting the four main ingredients.
Revised: We provide here an overview thereof, highlighting the four main ingredients.

Original: The third and fourth steps benefited from some inspiration drawn from the proof techniques of Agrawal and Devanur [2016].
Revised: The third and fourth steps beneﬁted from some inspiration drawn from the proof techniques of Agrawal and Devanur [2016].

Original: The first step is an adaptation of
Revised: The ﬁrst step is an adaptation of

Original: Lemmas 1 and 2 by Faury et al.
Revised: Lemmas 1 and 2 by Faury et al.

Original: First, the mentioned adaptation provides values of the parameters ε t ( a, x ) such that, with probability at least 1 − δ ,
Revised: First, the mentioned adaptation provides values of the parameters ε t ( a, x ) such that, with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: hence
Revised: hence

Revised: U

Original: [Equation]
Revised: [Equation]

Original: while (cid:88) t ⩽
Revised: while (cid:88) t (cid:54) T ε

Original: [Equation]
Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: , x

Revised: [Equation]

Original:  is of order √ T up to poly-logarithmic terms.
Revised: [Equation] = [Equation] null } is of order √ T up to poly-logarithmic terms.

Original: Second, the Phase 2 formulation of the strategy, in a primal form, is equivalently restated in a dual form.
Revised: Second, the Phase 2 formulation of the strategy, in a primal form, is equivalently restated in a dual form.

Original: For each round t ⩾ 2 , strong duality holds and entails the existence of a vector β budg ,⋆ t ∈ R d such that p t ( h t − 1 , · ) may be identified as the argmax over π : X → P ( A ) of
Revised: For each round t (cid:62) 2 , strong duality holds and entails the existence of a vector β budg ,(cid:63) t ∈ R d such that p t ( h t − 1 , · ) may be identiﬁed as the argmax over π : X → P ( A ) of

Original: E X ∼ ν
Revised: E X ∼ ν

Original: [Equation]
Revised: [Equation]

Original: x
Revised: a

Original: [Equation]
Revised: [Equation]

Original: (cid:35) .

Original: By exploiting the KKT conditions, we are able to get rid of the double sum above and finally get a
Revised: By exploiting the KKT conditions, we are able to get rid of the double sum above and ﬁnally get a

Original: X –pointwise characterization of p t ( h t − 1 , · ) : for all x ∈ X ,
Revised: X –pointwise characterization of p t ( h t − 1 , · ) : for all x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: Non-negative parts ( · ) + may be introduced thanks to the existence of the no-op action a null .
Revised: Non-negative parts ( · ) + may be introduced thanks to the existence of the no-op action a null .

Original: The distributions p t ( h t − 1 , x ) may therefore be interpreted as maximizing some upper-confidence bound on penalized gains (rewards minus some scalarized costs); the dual variables β budg ,⋆ t play a role similar to the Z parameter of Agrawal and Devanur [2016, Section 3.3] in terms of weighing gains versus costs.
Revised: The distributions p t ( h t − 1 , x ) may therefore be interpreted as maximizing some upper-conﬁdence bound on penalized gains (rewards minus some scalarized costs); the dual variables β budg ,(cid:63) t play a role similar to the Z parameter of Agrawal and Devanur [2016, Section 3.3] in terms of weighing gains versus costs.

Original: In passing, we also prove
Revised: In passing, we also prove

Original: [Equation]
Revised: [Equation]

Original: based on the KKT conditions.
Revised: based on the KKT conditions.

Original: The latter inequality is comparable in spirit to the bound of Agrawal and Devanur [2016, Corollary 3], relating Z to OPT ( ν, P, B ) /B .
Revised: The latter inequality is comparable in spirit to the bound of Agrawal and Devanur [2016, Corollary 3], relating Z to OPT ( ν, P, B ) /B .

Original: Third, for t ⩾ 2 , whenever the policy p t ( h t − 1 , · ) is obtained by solving the optimization problem
Revised: Third, for t (cid:62) 2 , whenever the policy p t ( h t − 1 , · ) is obtained by solving the optimization problem

Original: OPT ( ν, U t − 1 , B T ) of Phase 2 and by independence of x t and h t − 1 , we have
Revised: OPT ( ν, U t − 1 , B T ) of Phase 2 and by independence of x t and h t − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, repeated applications of the Hoeffding-Azuma inequality and the inequalities of the first step entail that, up to quantities of the order of √ T ,
Revised: Therefore, repeated applications of the Hoeffding-Azuma inequality and the inequalities of the ﬁrst step entail that, up to quantities of the order of √ T ,

Original: [Equation] a t , [Equation] x
Revised: T (cid:88)

Original: [Equation]

Original: y

Original: [Equation]
Revised: [Equation]

Original: We thus only need to control OPT ( ν, P, B ) −
Revised: We thus only need to control OPT ( ν, P, B ) −

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: OPT ( ν, U t − 1 , B T )
Revised: OPT ( ν, U t − 1 , B T )

Original: T , which may be assumed ⩾ 0 .
Revised: T , which may be assumed (cid:62) 0 .

Original: The value B T =
Revised: The value B T =

Original: B − 2 − (cid:112) 2 T ln(4 d/δ ) and similar Hoeffding-Azuma-based arguments show that with high probability, the budget limit B − 1 is indeed never reached and that we always compute p t ( h t − 1 , · ) in the way indicated by Phase 2.
Revised: B − 2 − (cid:112) 2 T ln(4 d/δ ) and similar Hoeffding-Azuma-based arguments show that with high probability, the budget limit B − 1 is indeed never reached and that we always compute p t ( h t − 1 , · ) in the way indicated by Phase 2.

Original: Fourth, we collect all bounds together.
Revised: Fourth, we collect all bounds together.

Original: We start with
Revised: We start with

Original: [Equation]
Revised: [Equation]

Original: We the exploit the dual characterization of p t ( h t − 1 , · ) and the control P ⩽ U t − 1 to get that with high probability, for all x ∈ X , 
Revised: We the exploit the dual characterization of p t ( h t − 1 , · ) and the control P (cid:54) U t − 1 to get that with high probability, for all x ∈ X , [Equation] ∈A

Original: [Equation]
Revised: [Equation]

Original: After integration over X ∼ ν and substituting of the definitions of π ⋆ and p t,a ( h t − 1 , · ) , as well as the equality stemming from the KKT conditions, we have
Revised: After integration over X ∼ ν and substituting of the deﬁnitions of π (cid:63) and p t,a ( h t − 1 , · ) , as well as the equality stemming from the KKT conditions, we have

Original: [Equation]
Revised: [Equation]

Original: ν
Revised: −

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] Rearranging and summing over 2 ⩽ t ⩽ T , we obtain
Revised: Rearranging and summing over 2 (cid:54) t (cid:54) T , we obtain

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: OPT [Equation]
Revised: OPT ( ν, P, B ) − OPT ( ν, U t − 1 , B T )

Original: T ⩽ [Equation] t [Equation]
Revised: T (cid:54) T (cid:88) t =

Original: B − B T
Revised: B − B T

Original: T [Equation] [Equation]
Revised: T ( β budg ,(cid:63) t ) T 1 (cid:54) (cid:18) B

Original: B T −(cid:19)
Revised: B T −(cid:19)

Original: OPT ( ν, P, B ) , where we substituted the first inequality stated in this fourth step.
Revised: OPT ( ν, P, B ) , where we substituted the ﬁrst inequality stated in this fourth step.

Original: This concludes the proof.
Revised: This concludes the proof.

Original: 4.2 Discussion onthe Main Technical Novelties
Revised: 4.2 Discussion on

Revised: the Main Technical Novelties

Original: As should be clear from the comments at the beginning of Section 4.1, the technical novelties lies in the second step of the proof of Theorem 1.
Revised: As should be clear from the comments at the beginning of Section 4.1, the technical novelties lies in the second step of the proof of Theorem 1.

Original: On the one hand, we are able to directly analyze a strategy stated in a primal form, which is a more natural formulation.
Revised: On the one hand, we are able to directly analyze a strategy stated in a primal form, which is a more natural formulation.

Original: On the other hand, doing so, we are also able to avoid the issues that come with dual formulations, relying, e.g., on some critical parameter
Revised: On the other hand, doing so, we are also able to avoid the issues that come with dual formulations, relying, e.g., on some critical parameter

Original: Z , as in Agrawal and Devanur [2016, Theorem 3], to trade off rewards and costs.
Revised: Z , as in Agrawal and Devanur [2016, Theorem 3], to trade off rewards and costs.

Original: This parameter Z should be of order OPT /B and has to be learned, e.g., through √ T initial exploration rounds.
Revised: This parameter Z should be of order OPT /B and has to be learned, e.g., through √ T initial exploration rounds.

Original: (More details are to be found in Section E.3.)
Revised: (More details are to be found in Section E.3.)

Original: In our analysis, this parameter Z is superseded by dual optimal variables β budg ,⋆ t ⩾ 0 , that are only used in the analysis and not to state the policy, unlike in Agrawal and Devanur [2016].
Revised: In our analysis, this parameter Z is superseded by dual optimal variables β budg ,(cid:63) t (cid:62) 0 , that are only used in the analysis and not to state the policy, unlike in Agrawal and Devanur [2016].

Original: Put differently, the clever use in this context of KKT conditions is the main technical novelty.
Revised: Put differently, the clever use in this context of KKT conditions is the main technical novelty.

Original: On a side note, we are also able to take care in an explicit and detailed fashion of the no-op action a null , whose specific treatment is often unaddressed in the literature.
Revised: On a side note, we are also able to take care in an explicit and detailed fashion of the no-op action a null , whose speciﬁc treatment is often unaddressed in the literature.

Original:  Analysis for an Unknown Context Distribution ν
Revised:  Analysis for an Unknown Context Distribution ν

Original: When the context distribution ν is unknown, we simply estimate it through its empirical frequencies (6).
Revised: When the context distribution ν is unknown, we simply estimate it through its empirical frequencies (6).

Original: The regret bound is almost unchanged: an additional mild factor of, e.g., 2 |X| (cid:112) 2 T ln(2 T |X| /δ ) appears in the √ T term multiplying OPT ( ν, P, B ) /B .
Revised: The regret bound is almost unchanged: an additional mild factor of, e.g., [Equation] (cid:112) 2 T ln(2 T |X| /δ ) appears in the √ T term multiplying OPT ( ν, P, B ) /B .

Original: This term comes from some uniform deviation argument stated in (7) and 8.
Revised: This term comes from some uniform deviation argument stated in (7) and 8.

Original: Theorem 2.
Revised: Theorem 2.

Original: In the setting of Box A of Section 2.1, we consider the adaptive policy of Box B of
Revised: In the setting of Box A of Section 2.1, we consider the adaptive policy of Box B of

Original: Section 3 with ˜ ν
Revised: Section 3 with ˜ ν

Original: = (cid:98) ν t at rounds t ⩾ 2 .
Revised: = (cid:98) ν t at rounds t (cid:62) 2 .

Original: We set a confidence level 1 − δ ∈ (0 , 1) and use parameters λ
Revised: We set a conﬁdence level 1 − δ ∈ (0 , 1) and use parameters λ

Original: = m ln(1 + T/m ) , a working budget of
Revised: = m ln(1 + T/m ) , a working budget of

Original: B [Equation] T , where
Revised: B − b T , where

Original: [Equation]
Revised: [Equation]

Original: and ε t ( a, x ) stated in (9) of the supplementary material.
Revised: and ε t ( a, x ) stated in (9) of the supplementary material.

Original: Then, provided that T ⩾ 2 m and B > 2 b T , we have, with probability at least 1 − 3 δ ,
Revised: Then, provided that T (cid:62) 2 m and B > 2 b T , we have, with probability at least 1 − 3 δ ,

Original: [Equation]
Revised: [Equation]

Original: where the expression of E T
Revised: where the expression of E T

Original: = O (cid:0) m √ T ln T (cid:1) may be found in (35) of the supplementary material.
Revised: = O (cid:0) m √ T ln T (cid:1) may be found in (35) of the supplementary material.

Original: The order of magnitude of the regret bound is (cid:0) m + |X| OPT ( ν, P, B ) /B (cid:1) √ T ln T , which is reminiscent of all known regret upper bounds for CBwK (e.g., the ones by Badanidiyuru et al.
Revised: The order of magnitude of the regret bound is (cid:0) m + |X| OPT ( ν, P, B ) /B (cid:1) √ T ln T , which is reminiscent of all known regret upper bounds for CBwK (e.g., the ones by Badanidiyuru et al.

Original: [2014] and Agrawal et al.
Revised: [2014] and Agrawal et al.

Original: [2016], for general CBwK, and Agrawal and Devanur [2016] for linear CBwK, see Section 6).
Revised: [2016], for general CBwK, and Agrawal and Devanur [2016] for linear CBwK, see Section 6).

Original: The factor |X| may be improved, see below, but this is a detail.
Revised: The factor |X| may be improved, see below, but this is a detail.

Original: A discussion on exhibiting corresponding lower bounds is to be found at the end of Section 6.
Revised: A discussion on exhibiting corresponding lower bounds is to be found at the end of Section 6.

Original: A detailed proof of Theorem 2 is provided in Appendix D of the supplementary material.
Revised: A detailed proof of Theorem 2 is provided in Appendix D of the supplementary material.

Original: It follows closely the proof of Theorem 1, with modifications mostly consisting of relating quantities of the form
Revised: It follows closely the proof of Theorem 1, with modiﬁcations mostly consisting of relating quantities of the form

Original: [Equation]
Revised: [Equation]

Original:  f ( X ) (cid:3) vs. E X ∼ ν (cid:2) f
Revised: , e.g. [Equation] f [Equation]

Original: ( X ) (cid:3) , where, e.g., f ( X) = (cid:88)
Revised: X

Original: [Equation]
Revised: [Equation]

Original: To do so, we use that for all functions f : X → [0 , 1] ,∀
Revised: To do so, we use that for all functions f : X → [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where (cid:119)(cid:119)(cid:98)
Revised: (7) where (cid:119)(cid:119)(cid:98)

Original: ν t − ν (cid:119)(cid:119) 1 is the total variation distance between (cid:98) ν t and ν .
Revised: ν t − ν (cid:119)(cid:119) 1 is the total variation distance between (cid:98) ν t and ν .

Original: In Appendix D, we upper bound the latter, for the sake of simplicity, in a crude way by applying T |X| times the Hoeffding-Azuma inequality (once for each 1 ⩽ t ⩽ T and x ∈ X ) and obtain that with probability at least 1 − δ , [Equation] ⩽ T,
Revised: In Appendix D, we upper bound the latter, for the sake of simplicity, in a crude way by applying T |X| times the Hoeffding-Azuma inequality (once for each 1 (cid:54) t (cid:54) T and x ∈ X ) and obtain that with probability at least 1 − δ , ∀ t (cid:54) T,

Original: [Equation]
Revised: [Equation]

Original: The |X| (cid:112) 2 T ln(2 T |X| /δ ) term in the regret bound of Theorem 2 appears as the sum over t ⩽ T of the deviation bounds (8).
Revised: The |X| (cid:112) 2 T ln(2 T |X| /δ ) term in the regret bound of Theorem 2 appears as the sum over t (cid:54) T of the deviation bounds (8).

Original: The bounds (8) may actually be improved into bounds of the order of (cid:112) |X| /t , via some Cauchy-Schwarz bound and a deviation argument in Banach spaces by Pinelis [1994], or by more direct techniques described by Devroye [1983, Lemma 3] and Berend and
Revised: The bounds (8) may actually be improved into bounds of the order of (cid:112) |X| /t , via some Cauchy-Schwarz bound and a deviation argument in Banach spaces by Pinelis [1994], or by more direct techniques described by Devroye [1983, Lemma 3] and Berend and

Original: Kontorovich [2012].
Revised: Kontorovich [2012].

Original: In any case, the regret bound of Theorem 2 automatically benefits from such improvements, by replacing the 2 |X| (cid:112) 2 T ln(2 T |X| /δ ) term in the bound therein by the (sum over t ⩽ T of the) better uniform deviation bounds.
Revised: In any case, the regret bound of Theorem 2 automatically beneﬁts from such improvements, by replacing the 2 |X| (cid:112) 2 T ln(2 T |X| /δ ) term in the bound therein by the (sum over t (cid:54) T of the) better uniform deviation bounds.

Original:  Extension to Linear Contextual Bandits with Knapsacks
Revised:  Extension to Linear Contextual Bandits with Knapsacks

Original: This section is a brief summary of Appendix E. We explain therein how the adaptive policy of Box B may be adapted to the setting of linear CBwK, introduced by Agrawal and Devanur [2016], where the bounded rewards r t and vector costs c t are independently generated at each round according to bounded distributions with respective expectations r ( a t , x t ) and c ( a t , x t ) , depending linearly on (a transfer function φ of) the contexts: for all a ̸ = a null and x ∈ X , for all components i of c , r ( a, x ) = 
Revised: This section is a brief summary of Appendix E. We explain therein how the adaptive policy of Box B may be adapted to the setting of linear CBwK, introduced by Agrawal and Devanur [2016], where the bounded rewards r t and vector costs c t are independently generated at each round according to bounded distributions with respective expectations r ( a t , x t ) and c ( a t , x t ) , depending linearly on (a transfer function ϕ of) the contexts: for all a (cid:54) = a null and x ∈ X , for all components i of c , r ( a, x ) = ϕ

Original: φ ( a, x ) T µ ⋆
Revised: ( a, x ) T µ (cid:63) and

Original: and c i ( a, x ) =
Revised: c i ( a, x ) =

Original: φ ( a, x ) T θ ⋆,i .
Revised: ϕ ( a, x ) T θ (cid:63),i .

Original: We consider the same benchmark OPT ( ν, r, c , B ) as Agrawal and Devanur [2016] and are able to exhibit a similar (cid:0) OPT ( ν, r, c , B ) /B
Revised: We consider the same benchmark OPT ( ν, r, c , B ) as Agrawal and Devanur [2016] and are able to exhibit a similar (cid:0) OPT ( ν, r, c , B ) /B

Original: (cid:1) m √ T ln T regret bound, with however a slight relaxation on the order of magnitude required for B .
Revised: (cid:1) m √ T ln T regret bound, with however a slight relaxation on the order of magnitude required for B .

Original: We do so with a strategy that we deem more direct and natural, inspired from the one of Box B, where in Phase 1 a LinUCB-type (Abbasi-Yadkori et al.
Revised: We do so with a strategy that we deem more direct and natural, inspired from the one of Box B, where in Phase 1 a LinUCB-type (Abbasi-Yadkori et al.

Original: [2011]) estimation of the parameters is performed, and where in Phase 2, a direct solution to an OPT problem with estimated parameters is performed.
Revised: [2011]) estimation of the parameters is performed, and where in Phase 2, a direct solution to an OPT problem with estimated parameters is performed.

Original: The parameters are upper-confidence functions U t − 1 on r and lower-confidence vector functions L t − 1 on c .
Revised: The parameters are upper-conﬁdence functions U t − 1 on r and lower-conﬁdence vector functions L t − 1 on c .

Original: The main advantage of our approach in the case of linear contextual bandits is exactly as described in
Revised: The main advantage of our approach in the case of linear contextual bandits is exactly as described in

Original: Section 4.2: avoiding the critical parameter Z of Agrawal and Devanur [2016, Theorem 3], which is used to trade off rewards and costs.
Revised: Section 4.2: avoiding the critical parameter Z of Agrawal and Devanur [2016, Theorem 3], which is used to trade off rewards and costs.

Original: The main limitation of our approach is the assumption of a finite context set X , which is required to make the Phase-2 linear program tractable.
Revised: The main limitation of our approach is the assumption of a ﬁnite context set X , which is required to make the Phase-2 linear program tractable.

Original:  Future Work
Revised:  Future Work

Original: We conclude this article with a list of issues to be further investigated.
Revised: We conclude this article with a list of issues to be further investigated.

Original: First , as discussed in Section 2.2, the restrictions of finiteness should be alleviated: finiteness of the context set in the setting of this article, or finiteness of the set of benchmark policies in other settings (see Badanidiyuru et al., 2014 and Agrawal et al., 2016).
Revised: First , as discussed in Section 2.2, the restrictions of ﬁniteness should be alleviated: ﬁniteness of the context set in the setting of this article, or ﬁniteness of the set of benchmark policies in other settings (see Badanidiyuru et al., 2014 and Agrawal et al., 2016).

Original: Second , we only dealt with ⩽ budget constraints (and do does the literature so far).
Revised: Second , we only dealt with (cid:54) budget constraints (and do does the literature so far).

Original: Direct approaches to constraints of the form ⩾ remain to be further investigated.
Revised: Direct approaches to constraints of the form (cid:62) remain to be further investigated.

Original: A third series of questions to be clarifies concerns regret lower bounds, and more generally, the tightness of the results—in particular, the required conditions on budget sizes.
Revised: A third series of questions to be clariﬁes concerns regret lower bounds, and more generally, the tightness of the results—in particular, the required conditions on budget sizes.

Original: Earlier references for contextual bandits with knapsacks did also not provide lower bounds statements that were simultaneously optimal (i.e., matching the obtained upper bounds) and general (i.e., valid for all problems with a given number of rounds T , a given budget B , and a given value OPT for the optimal expected reward achievable by a static policy).
Revised: Earlier references for contextual bandits with knapsacks did also not provide lower bounds statements that were simultaneously optimal (i.e., matching the obtained upper bounds) and general (i.e., valid for all problems with a given number of rounds T , a given budget B , and a given value OPT for the optimal expected reward achievable by a static policy).

Original: Badanidiyuru et al.
Revised: Badanidiyuru et al.

Original: [2014, comments after Theorem 1] merely indicates that the obtained regret upper bound is optimal in some regimes, e.g., when the budget B grows linearly with the number of rounds T .
Revised: [2014, comments after Theorem 1] merely indicates that the obtained regret upper bound is optimal in some regimes, e.g., when the budget B grows linearly with the number of rounds T .

Original: Agrawal and Devanur [2016, comments after
Revised: Agrawal and Devanur [2016, comments after

Original: Theorem 1] only compares the obtained regret upper bound to the case of no budget constraints.
Revised: Theorem 1] only compares the obtained regret upper bound to the case of no budget constraints.

Original: In particular, as far as the orders of magnitude in T are concerned, it is unclear whether the ( OPT /B ) √ T rates achieved (up to poly-logarithmic factors) in the present article and in the two mentioned references are optimal.
Revised: In particular, as far as the orders of magnitude in T are concerned, it is unclear whether the ( OPT /B ) √ T rates achieved (up to poly-logarithmic factors) in the present article and in the two mentioned references are optimal.

Original: These rates do not match the optimal rates in the case of no contexts, which were stated and proved by Badanidiyuru et al.
Revised: These rates do not match the optimal rates in the case of no contexts, which were stated and proved by Badanidiyuru et al.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Zhen Li and Gilles Stoltz have no direct funding to acknowledge other than the salaries paid by their employers, BNP Paribas, CNRS, and HEC Paris.
Revised: Zhen Li and Gilles Stoltz have no direct funding to acknowledge other than the salaries paid by their employers, BNP Paribas, CNRS, and HEC Paris.

Original: They have no competing interests to declare.
Revised: They have no competing interests to declare.