Analyzing Thompson Sampling for Contextual Bandits via the Lifted Information Ratio Anonymous Author(s) AfﬁliationAddress email
Abstract
We study the Bayesian regret of the renowned Thompson Sampling algorithmin contextual bandits with binary losses and adversarially-selected contexts.
Weadapt the information-theoretic perspective of Russo and Van Roy [2016] to thecontextual setting by considering a lifted version of the information ratio deﬁnedin terms of the unknown model parameter instead of the optimal action or optimalpolicy as done in previous works on the same setting.
This allows us to boundthe regret in terms of the entropy of the prior distribution through a remarkablysimple proof, and with no structural assumptions on the likelihood or the prior.
The extension to priors with inﬁnite entropy only requires a Lipschitz assumptionon the log-likelihood.
An interesting special case is that of logistic bandits withd -dimensional parameters, K actions, and Lipschitz logits, for which we provide a(cid:101) O ( √ dKT ) regret upper-bound that does not depend on the smallest slope of thesigmoid link function.
Introduction
Thompson sampling is one of the most popular algorithms for sequential decision making underuncertainty.
First proposed by Thompson [1933], it has been rediscovered several times over theconsequent decades and has been eventually popularized in the machine learning literature by
Chapelle and Li [2011] and Scott [2010] who pointed out its excellent empirical performancefor solving contextual bandit problems.
These empirical studies were followed by a sequence ofbreakthroughs on the front of theoretical analysis, spearheaded by the works of Agrawal and Goyal[2012, 2013a,b], Kaufmann et al.
[2012] and Russo and Van Roy [2014], Russo and Van Roy [2016].
Thanks to these successes, Thompson sampling has become one of the gold-standard methods forsolving multi-armed bandit problems.
Indeed, in the last decade, several Thompson-sampling-stylemethods have been developed and analyzed for a variety of problem settings.
The variety of different analysis techniques applied to Thompson sampling is perhaps even largerthan the variety of problem settings that it has been applied to.
The ﬁrst key tools for analyzing the
Bayesian regret of Thompson sampling for multi-armed bandits have been developed by Russo and
Van Roy [2014], and our analysis naturally borrows several of these tools.
The worst-case resultsdeveloped in said work were reﬁned in [Russo and Van Roy, 2016], where they proved for the ﬁrsttime “information-theoretic” bounds on the regret of TS that scale with the Shannon entropy ofthe optimal action A ∗ under the prior on the model parameters.
This result has inspired a rangeof follow-up works, including extensions to uncountable action sets [Dong and Van Roy, 2018],approximate implementations [Lu and Van Roy, 2017, Qin et al., 2022], and even new algorithmsbased on the analysis technique itself [Russo and Van Roy, 2018, Kirschner and Krause, 2018].
Onelimitation that this technique could not overcome so far is not being able to satisfyingly deal withcontext .
When considering i.i.d.
contexts and ﬁnite policy classes, one can apply the theory of Russo and
Van Roy [2016] and treat policies as actions (as done by Liu et al. 2018, Bubeck and Sellke 2020),or adapt the information-ratio directly (as done by Kirschner et al. 2020, Hao et al. 2022) to obtainregret bounds scaling with the entropy of the optimal policy. This approach, however, can lead toa polynomial dependence on the number of contexts when assuming no further structure about therewards, as the entropy of the optimal policy may be as large as log | Π | = |X| log |A| .
Another variantof Thompson sampling demonstrating a similar prior-dependent regret bound has been proposed by Li[2013], whose regret guarantees also suffer from a suboptimal dependence on the number of rounds
T .
A much more satisfying solution has been recently given by Zhang [2021], whose “feel-good
Thompson sampling” method guarantees both frequentist and Bayesian regret bounds of the order 46 √
KT log N , where K is the number of actions, T the number of rounds, and N the support size ofthe prior distribution on model parameters θ (cid:63) .
Under a Lipschitzness assumption on the prior andthe likelihood, Zhang proves a frequentist regret bound scaling with the log-prior-probability massassigned to the true parameter θ (cid:63) .
The techniques involved in proving these results drew substantialinspiration from the works of Foster and Rakhlin [2020], Foster and Krishnamurthy [2021] and infact results of a similar ﬂavor were also proved recently in Foster et al.
Our own approach can be seen as a reconciliation of the analysis style of Zhang [2021] with theinformation-theoretic methodology of Russo and Van Roy [2016].
Our main conceptual contributionis proposing an adjustment of the now-classic notion of “information ratio” proposed by Russo and
Van Roy [2016] that applies to contextual bandits.
In its original deﬁnition, the information ratioquantiﬁes the tradeoff between incurring low regret and gaining information about the optimal action.
As we will argue, this notion of information gain is inappropriate for contextual bandits with noni.i.d.
contexts. We propose a variant that measures the amount of information gained about the truemodel parameter θ (cid:63) instead of the optimal action (which may be context dependent).
The complexitynotion resulting from this extension is called the “lifted information ratio”.
Our analysis shows thatthe Bayesian regret of Thompson sampling can be bounded in terms of the lifted information ratioand the Shannon entropy of the hidden parameter θ (cid:63) , which mirrors the result of Russo and Van Roy[2016].
Along the way, we draw inspiration from the recently proposed analysis technique of Zhang[2021] for contextual bandits, and in fact we show that our notion of lifted information ratio bridgesthe concept of “decoupling coefﬁcient” proposed by Zhang with the information ratio of Russo and Van Roy.
We state our main results in the context of K -armed contextual bandits with binary losses.
Forcountable parameter spaces, we prove that the Bayesian regret of Thompson sampling satisﬁes abound of order (cid:112) KTH ( θ ∗ ) , where H denotes the Shannon entropy.
This result is comparableto the bound of Foster and Krishnamurthy [2021] for the FastCB algorithm, which is of the order 71 (cid:112)
KT log | Θ | and holds in a frequentist sense.
This is the best result we are aware of for this setting.
To demonstrate the ﬂexibility of our technique, we provide an extension to logistic bandits with
Lipschitz continuous logits, generalizing the well-studied setting involving logits that are linearfunctions of the context and the parameter θ .
For this setting, we prove a regret bound of order 75 (cid:112)
KT log N 1 /CT
(Θ , (cid:107)·(cid:107) ) , where N ε (Θ , (cid:107)·(cid:107) ) is the ε -covering number of Θ under norm (cid:107)·(cid:107) and Cis the Lipschitz constant of the logits.
This implies a regret bound of order √ KdT in the well-studiedcase of linear logits.
Notably, the bound does not show any dependence on the smallest slope κ of thesigmoid link function that almost all existing results for this setting suffer from [Filippi et al., 2010,
Kveton et al., 2020, Abeille et al., 2021, Faury et al., 2022].
Indeed, this constant has plagued all regretbounds since the early work of Filippi et al.
[2010] and was only recently moved to lower-order termsby the breakthrough work of Abeille et al.
Bounds involving other potentially large problemdependent constants have also been proved in the Bayesian setting by Dong and Van Roy [2018] and
Dong et al.
To our knowledge, our bounds are the ﬁrst to entirely remove this factor.
The rest of the paper is organized as follows.
After introducing the necessary technical backgroundin Section 2, we discuss matters of information gains, information ratios, and decoupling coefﬁcientsin Section 3.
We state our main results and instantiate them in a variety of settings in Section 4.
Weprovide the key ideas of the analysis in Section 5 and conclude in Section 6.
Notation.
For a natural number n ∈ N , [ n ] = { 1 , 2 , . . . , n } denotes the set of the ﬁrst n naturalnumbers.
For x, y ∈ R d , (cid:104) x, y (cid:105) denotes the canonical scalar product of x and y , and (cid:107) x (cid:107) the Euclideannorm of x .
We denote the Shannon entropy of a discrete random variable Z with probability massfunction P as H ( Z ) =
− E [log( P ( Z ))] .
We use ¯0 d for a d -dimensional vector of zeros and I d forthe d × d identity matrix.
Preliminaries
We consider a parametric class of contextual bandits with parameter space Θ , context space X , and
K actions.
To each parameter θ ∈ Θ there corresponds a contextual bandit with loss distribution
P θ,x,a for each context x ∈ X and action a ∈ A , with the mean of the loss distribution denoted by(cid:96) ( θ, x, a ) .
We will dedicate special attention to the case where the losses are binary and thus P θ,x,a isa Bernoulli distribution with parameter (cid:96) ( θ, x, a ) .
For the main part of our theoretical analysis, wewill assume Θ is either a ﬁnite set or a bounded metric space.
We study the problem of regret minimization in the Bayesian setting.
In this setting, the environmentsecretly samples a parameter θ (cid:63) from a known prior distribution Q 1 over Θ .
We assume that theagent has full knowledge of the prior and the likelihood model P θ,x,a .
The agent interacts withthe environment for T rounds as follows.
At each round t ∈ [ T ] , an adaptive adversary selects acontext X t , possibly using randomization and taking into account the previous history of actions andlosses, but not θ (cid:63) .
After observing the context X t , the agent selects an action A t ∈ A (possibly usingrandomization) and incurs a binary loss L t ∼ P θ (cid:63) ,X t ,A t .
The goal of the agent is to minimize theexpected sum of losses.
In the Bayesian setting, this is equivalent to minimizing the Bayesian regret,deﬁned as follows:
[Equation]
where A (cid:63)t = arg max a ∈A (cid:96) ( θ (cid:63) , X t , a ) is the optimal action for round t , and the expectation in (1) isover all sources of randomness: the initial sampling of θ (cid:63) from Q 1 , the agent’s randomization overactions, and the randomness of the loss realizations.
Furthermore, let F t = σ ( X 1 , A 1 , L 1 , . . .
, X t , A t , L t ) be the sigma-algebra representing the historyof contexts, actions and losses observed by the agent up to time t included.
We use Q t to denote thedistribution of the unknown parameter θ (cid:63) conditional on the past history F t − 1 , and simply call it theposterior distribution.
We denote by π t ( ·| X t ) the distribution over the agent’s actions conditional on
X t and F t − 1 , and call it the agent’s policy.
Finally, we will frequently use the shorthand notations 117 E t [ · ] = E [ · |F t − 1 , X t ] and P t [ · ] = P [ · |F t − 1 , X t ] .
This paper is dedicated to the study of the celebrated Thompson Sampling (TS) algorithm, deﬁnedas follows.
At each round t , TS draws a parameter θ t from the posterior distribution Q t .
Then, itselects the action that maximizes (cid:96) ( θ t , X t , · ) .
Finally, it updates Q t via Bayes’ rule, obtaining the newposterior Q t +1 .
The algorithm can be equivalently deﬁned as a method that plays actions accordingto their posterior probability of being optimal, that is: P [ A t = a |F t − 1 , X t ]
= P
[ A (cid:63)t = a |F t − 1 , X t ] .
The pseudocode is shown as Algorithm 1.
Algorithm
Regret, information ratio, and decoupling coefﬁcient
The classic results of Russo and Van Roy [2016] establish that the regret of Thompson sampling innon-contextual multi-armed bandit problems can be upper bounded in terms of a quantity called the information ratio .
Informally, the information ratio measures the tradeoff between achieving lowregret and gaining information about the identity of the optimal action A (cid:63) (which is a deterministicfunction of θ (cid:63) in the standard multi-armed bandit setting).
The formal deﬁnition is given by
[Equation]
where I t ( A (cid:63)t ; ( A t , L t )) denotes the mutual information between A (cid:63) and the action-observation pair( L t , A t ) , conditioned on the history F t − 1 .
Intuitively, having small information ratio implies thatevery time Thompson sampling suffers large regret, it has to gain a lot of information about theoptimal action, which suggests that it should be possible to bound the total regret by the total amountof information that there is to be gained.
The result of Russo and Van Roy [2016] conﬁrms thisintuition by showing that the regret of Thompson sampling is of the order (cid:112) ρ ∗ TH
( A (cid:63) ) , whereρ ∗ = max t ρ ∗ t and H is the Shannon entropy.
The information ratio itself can always be upperbounded by K/ 2 , but better bounds can be shown when the loss function has favorable structuralproperties (e.g., whenever the reward function is a d -dimensional linear function, the informationratio is at most d/ 2 ).
While this result and the underlying information-theoretic framework is very elegant, it is inappropriate for studying contextual bandit problems.
The speciﬁc challenge is that the optimal action A (cid:63)tchanges from round to round and gaining large amount of information about A (cid:63)t for any given roundmay not necessarily be useful for predicting future actions.
To see this, consider a stylized examplewith action set { 1 , 2 , . .
, K } , where there exists an action a reveal
= 1 whose loss entirely revealsthe identity of the optimal action a (cid:63) ( x ) for context x : (cid:96) ( θ (cid:63) , x, a reveal )
= 1 − 10 − a (cid:63) ( x ) .
Taking thisaction provides large information gain about A (cid:63)t , but results in large regret and reveals nothing aboutthe future losses.
Thus, in the contextual setting, one can keep following a policy that provides lowinformation ratio while suffering linear regret.
This issue necessitates an alternative deﬁnition thatstill permits an effective information-theoretic analysis.
Our proposition is to consider a relaxed deﬁnition of the information ratio based on the mutualinformation between the true parameter θ (cid:63) and the observed loss.
In particular, we deﬁne
[Equation]
where I t ( θ (cid:63) ; L t ) =
E t (cid:2) D KL (cid:0) P
[Equation]
(cid:63) [Equation] [Equation] (cid:13)(cid:13) P
[Equation] 1 ,X t ,A t (cid:1)(cid:3) is the mutual information be-
153 tween θ (cid:63) and L t , conditioned on the history F t − 1 and the context-action pair X t , A t .
This quantitymeasures the information that the agent gains about θ (cid:63) .
High values of I t ( θ (cid:63) ; L t ) intuitively allowmaking better predictions about the future loss realizations for all possible context sequences.
Since
A (cid:63)t is a deterministic function of θ (cid:63) given X t , the data processing inequality implies that the information gain about θ (cid:63) is always greater than that about A (cid:63)t , which in turn implies that ρ t is smaller thanwhat one would obtain by directly generalizing the deﬁnition of Russo and Van Roy [2016].
As thisnotion of information gain measures the efﬁciency of inferring the identity of a hidden parameter,we refer to I t ( θ (cid:63) ; L t ) as the lifted information gain , and ρ t as deﬁned in Equation (2) as the liftedinformation ratio.
As our analysis will establish, a bounded lifted information ratio guarantees lowregret, and we will show that the ratio itself can be bounded reasonably under conditions similar tothe ones required by the analysis of Russo and Van Roy [2016], despite the fact that all quantities aredeﬁned conditionally on a sequence of non-i.i.d. contexts.. We are not the ﬁrst to consider the lifted information ratio: very similar quantities have beenconsidered in the literature on information-directed sampling algorithms, as initiated by Russo and Van Roy [2018], who have already introduced an the non-contextual counterpart of the same quantityin their Section 9.2.
The recent works of Kirschner et al.
[2020], Hao et al. [2022] also considerversions of the standard information ratio for contextual bandits, but their analysis crucially relies onthe assumption that the contexts are i.i.d. Similarly, Lu et al. [2021], Liu et al.
[2022] deﬁne versionsof the information ratio using various targets for information-directed sampling, which recovers ourdeﬁnition of the lifted information ratio when setting the target as θ (cid:63) .
While these works do notdistinguish between all these versions of the information ratio on the level of terminology, we believethat emphasizing the “lifted” nature of our information ratio provides more clarity in that it indicatesthe speciﬁc features of contextual bandit learning with parametric losses in a transparent way.
Thus,we will keep using this terminology throughout the paper.
Our lifted information ratio is also closely related to a quantity appearing in the analysis of Zhang[2021], called the “decoupling coefﬁcient”.
Adapted to our Bayesian setting, this coefﬁcient can bedeﬁned as the smallest constant δ such that the following inequality holds:
[Equation]
where the ﬁrst line gives the original deﬁnition mirroring that of Zhang [2021] and the second lineplugs in the choice of µ achieving the inﬁmum.
Reordering gives the value of the optimal δ ∗ t :
[Equation]
which matches our deﬁnition of the lifted information ratio, up to the difference of replacing themutual information by the root mean-squared error in predicting the true parameter θ ∗ .
Notably, thisdeﬁnition essentially coincides with the lifted information ratio for the special case of Gaussian losses.
Main results
In this section, we state our main results concerning the Bayesian regret of Thompson sampling forcontextual bandits.
We will assume that the losses are binary and the action space is ﬁnite, unlessotherwise stated.
However, several of our results can be generalized beyond this setting.
We willillustrate this in Section 4.3, where we provide some additional results for the classic setting of
Gaussian linear contextual bandits.
We begin by stating two general regret bounds in terms of the lifted information ratio deﬁned in
Equation (2).
The reader that is not interested in the full generality of our theory may skip to
Section 4.2 for concrete regret bounds.
Our ﬁrst abstract bound applies to priors with ﬁnite entropy,the simplest example being ﬁnite parameter spaces.
Theorem 1.
Assume Q 1 is supported on the countable set Θ 1 ⊆ Θ and that the lifted informationratio for all rounds t satisﬁes ρ t ≤ ρ for some ρ > 0 .
Then, the Bayesian regret of TS after T roundscan be bounded as
[Equation]
In particular if Θ 1 is a ﬁnite set with | Θ 1 | =
N , the regret of TS satisﬁes
[Equation]
The proof of this theorem is stated in Section 5.1.
Unfortunately, the Shannon entropy can beunbounded for distributions with inﬁnite support, which is in fact the typical situation that oneencouters in practice.
To address this concern, we develop a more general result, that holds for abroader family of distributions.
In the following, (Θ , (cid:37) ) is a metric space with metric (cid:37) :
Θ 2 → R .
We make the following regularity assumption on the likelihood function P θ,x,a :
Assumption 1.
There exists a constant C > 0 such that for any θ, θ (cid:48)
∈ Θ 1 , 205 | log P θ,x,a ( L ) − log P θ (cid:48) ,x,a ( L ) | ≤ C(cid:37)
( θ, θ (cid:48) ) holds for all x ∈ X , a ∈ A , and L ∈ { 0 , 1 } .
Under this assumption, we can state a variant of Theorem 1 that applies to metric parameter spaces:
Theorem 2.
Assume (Θ , (cid:37) ) is a metric space, and Q 1 is supported on Θ 1 ⊆ Θ with ε -coveringnumber N ε
(Θ 1 , (cid:37) ) .
Let Assumption 1 hold, and assume the lifted information ratio for all rounds tsatisﬁes ρ t ≤ ρ for some ρ > 0 .
Then, the Bayesian regret of TS after T rounds can be bounded as
[Equation]
The proof is based on a covering argument on top of the proof of the previous theorem, and isprovided in Appendix A.2.
To get a better understanding of Assumption 1, it is useful to notice that itis satisﬁed in basic settings, like logistic bandits with Lipschitz logits.
See Section 4.2 for details.
4.1 Bounding the lifted information ratio
At this point, some readers may worry that the lifted information ratio may be impossible to bounddue to the lifting to the space of parameters Θ .
To address this concern, we now turn to showingbounds on the lifted information ratio.
We ﬁrst consider the unstructured case, that holds for arbitraryparameter spaces, likelihoods, and priors.
Lemma 1.
Suppose that the losses are binary and |A| =
K .
Then, the lifted information ratio of
Thompson sampling satisﬁes ρ t ≤ 2 K for all t ≥ 1 .
The proof of this lemma (provided in Section 5.2) relies on a decoupling argument between the choiceof the action and of the parameters at round t , inspired by Zhang [2021].
Taking his argument onestep further, we center our analysis around an application of convex conjugacy, which we believemay be applicable in a broader variety of settings.
We wish to highlight that this proof technique isvery different from the information-theoretic methodology pioneered by Russo and Van Roy [2016].
Next, we consider the case of linear expected losses in Euclidean parameter spaces, which, inprinciple, allows for an unbounded number of actions.
Lemma 2.
Suppose that Θ ⊆ R d , the losses are binary, and the expected losses are linear functionsof the form (cid:96) ( θ, x, a )
= (cid:104) θ,
ϕ ( x, a ) (cid:105) , where ϕ :
X × A → R d is a feature map, such that (cid:96) ( θ, x, a ) ∈[0 , 1] for all x ∈ X , a ∈ A , θ ∈ Θ .
Then, the lifted information ratio satisﬁes ρ t ≤ d/ 2 .
The proof of this result (deferred to Appendix B) follows the arguments of Proposition 5 of Russoand Van Roy [2016], with some small but important changes to account for the presence of contexts.
Notably, both of these results match the classic bounds of Russo and Van Roy [2016] on the standarddeﬁnition of the information ratio for these settings (cf.
their Propositions 5 and 3), implying thatlifting to the space of parameters does not substantially impact the regret-information tradeoff.
4.2 Concrete regret bounds for Bernoulli bandits
We now instantiate our bounds in two well-studied settings for Bernoulli bandits.
We start from thefully unstructured case, assuming ﬁnite actions and ﬁnitely supported prior.
The following regretbound follows directly from Theorem 1 and Lemma 1.
Theorem 3.
Consider a contextual bandit with K actions and binary losses, and suppose Θ 1 , thesupport of Q 1 , is ﬁnite with | Θ 1 | = N .
Then, the Bayesian regret of TS satisﬁes:
[Equation]
This result is comparable to the best known regret guarantees for this problem due to Foster and
Krishnamurthy [2021] and Zhang [2021], and matches the minimax rate for unstructured contextualbandits with a policy class of size N [Beygelzimer et al., 2011, Dudík et al., 2011].
However, weare not aware of a comparable result for the Thompson sampling algorithm in the literature, be it
Bayesian or not.
Moving to Bernoulli bandits with structure, we consider a well-studied setting known as logistic bandits .
In this model, the losses are generated by a Bernoulli distribution as L t ( θ, x, a ) ∼
Ber( σ ( f θ ( x, a ))) , where σ ( z ) =
1 / (1 + e − z ) is the sigmoid function.
We just assume that f θ ( x, a ) ,called the logit function, is C -Lipschitz in θ , which directly implies that Assumption 1 holds.
Noticethat our deﬁnition generalizes the commonly used notion of logistic bandits that consider linear logitfunctions of the form f θ ( x, a ) = (cid:104) θ, ϕ ( x, a ) (cid:105) , where ϕ :
X × A → R d is some feature map.
Ourresult for logistic bandits is based on Theorem 2 and Lemma 1.
Theorem 4.
Assume Θ ⊂ R d and (cid:107) θ (cid:107) ≤ S for all θ ∈ Θ 1 .
Consider a class of logistic bandits with
K actions and C -Lipschitz logit function f θ ( x, a ) .
Then, the Bayesian regret of TS after T roundscan be bounded as:
[Equation]
The proof follows from an application of Theorem 2, and can be found in Appendix A.3.
To ourknowledge, this is the ﬁrst regret bound for logistic bandits with nonlinear logits.
We can furtherspecialize our result to linear logits, the setting that is most commonly studied in the literature:
Corollary 1.
Assume Θ ⊂ R d , (cid:107) θ (cid:107) ≤ S for all θ ∈ Θ 1 .
Consider a class of logistic bandits with
K actions and linear logit function f θ ( x, a ) = (cid:104) ϕ
( x, a ) , θ (cid:105) , with (cid:107) φ ( x, a ) (cid:107) ≤ B for all x ∈ X anda ∈
[ K ] .
Then, the Bayesian regret of TS after T rounds can be bounded as:
[Equation]
The proof of the corollary is stated at Appendix A.3.
A remarkable feature of this bound is that itshows no dependence on the minimum derivative of the sigmoid link function, albeit at the priceof a √ K factor in the bound.
Nevertheless, we believe this to be the ﬁrst regret guarantee thatentirely gets rid of this potentially enormous constant without very strong assumptions.
Indeed,this constant has been present in nearly all previous bounds we are aware of [Filippi et al., 2010,
Li et al., 2017, Faury et al., 2020, Abeille et al., 2021, Faury et al., 2022]—although these resultshave the advantage of holding in a frequentist sense.
In the Bayesian setting, the works of Dong and
Van Roy [2018] and Dong et al.
[2019] have proved a variety of bounds on the regret of Thompsonsampling for non-contextual logistic bandits, but none of them are directly comparable with our resultabove.
Dong et al.
[2019] prove a regret bound of order d √ T for a highly specialized setting with
A = Θ , and a range of other bounds under a variety of strong assumptions.
An improved bound forthe many-actions setting that scales at most logarithmically with K remains an open problem.
Itsdifﬁculty is testiﬁed by a set of negative examples provided by Dong et al.
[2019], and by a long-livedconjecture of [Dong and Van Roy, 2018]
regarding the information ratio for logistic bandits, that, toour knowledge, has not yet been veriﬁed in theory.
4.3 Beyond binary losses: linear bandits with Gaussian noise
We now illustrate how our techniques (in particular the lifted information ratio) can be extendedbeyond the case of binary losses, and in particular consider the classic setting of Bayesian linearcontextual bandits, where the loss is a linear function of a d -dimensional feature map with additive
Gaussian noise, and the prior is also Gaussian.
Lemma 3.
Suppose θ ∗ be a d -dimensional normal random vector, so θ ∗ ∼ N (¯0 d , λI d ) for someλ > 0 , the loss is a normal random variable L t ∼ N ( (cid:96) ( θ (cid:63) , X t , A t ) , σ 2 ) for some standard deviationσ > 0 , and the expected loss is linear, so (cid:96) ( θ, x, a ) =
(cid:104) θ,
ϕ ( x, a ) (cid:105) , where ϕ : X ×
[ K ] → R d is afeature map.
Then, the lifted information ratio satisﬁes ρ t ≤ min { d, 2(1 + log K ) } .
For this setting, the bound ρ t ≤ d has already been shown for the classic information ratio by
Russo and Van Roy [2016].
However, we believe that our bound ρ t ≤ 2(1 + log K ) is new forany deﬁnition of information ratio.
By combining this bound on the lifted information ratio withstandard arguments for linear contextual bandits, we recover both of the well-known regret bounds oforder d √ T log T and (cid:112) dT log ( KT ) for this seting, respectively due to Abbasi-Yadkori et al.
[2011]andChu et al.
2 See Corollary 2 in Appendix B for a rigorous statement and the proof.
Thisresult, although not surprising, indicates once again that our notion of lifted information ratio doesnot lead to compromises in performance, even when the losses are not binary.
Analysis
This section presents the key ideas of the proofs of our main results.
We will just provide the proof of
Theorem 1 and that of Lemma 1, which we believe offer the most insight into our techniques.
Allother proofs, included those of auxiliary lemmas, are deferred to Appendix A and B. For sake ofclarity, we focus on the relatively simple case where Θ 1 is countable, so that (with a slight abuse ofnotation) we can write Q t ( θ ) to denote the posterior probability associated with θ .
Note, however,that our full proofs also handle the case of general distributions (details in Appendix A.1).
5.1 The proof of Theorem 1
Recalling the deﬁnition of the lifted information ratio (Equation 2), we ﬁrst notice that the regret canbe rewritten as follows:
[Equation]
where the ﬁrst step uses the tower rule of expectation, the second step the deﬁnition of ρ t , and theﬁnal step follows from the Cauchy–Schwarz inequality.
The key challenge is then to bound the sum of information-gain terms.
The following lemma providesa more tractable form of this sum:
Lemma 4.
Under the assumptions of Theorem 1,
[Equation]
The proof of this lemma is based on a classic “Bayesian telescoping” argument that we have learnedfrom Grünwald [2012].
We provide the proof of Lemma 4 in Appendix A.1.
Supposing now that theprior has bounded entropy, we can easily bound the term appearing on the right hand side as follows:
[Equation]
This concludes the proof of the ﬁrst statement.
The second statement follows from the ﬁrst using thetrivial bound on the Shannon entropy of any ﬁnite-support distribution.
5.2 The proof of Lemma 1
We start by introducing some notation that will be useful for the proof.
In particular, we use g todenote the binary relative entropy function deﬁned for all p, q ∈ [0 , 1] as
[Equation]
and we use the convention 0 · log 0 = 0 .
Furthermore, we deﬁne the posterior mean loss as(cid:96) t ( x, a ) =
E t [ (cid:96) ( θ (cid:63) , x, a )] .
These notations allow us to conveniently rewrite the information gain as
I t ( θ (cid:63) ; L t ) =
[Equation]
A t (cid:13)(cid:13)
P L t |F t − 1 ,X t ,A t (cid:1)(cid:3) =
E t (cid:2) g (cid:0) (cid:96) ( θ (cid:63) , X t , A t ) (cid:13)(cid:13) (cid:96) t
( X t , A t ) (cid:1)(cid:3) .
We will now prove a generalization of Lemma 1, which will directly imply the original result:
Lemma 5.
Under the assumptions of Lemma 1, for all t , the lifted information ratio of Thompsonsampling satisﬁes ρ t ≤ 2 (cid:80) a ∈A E t [Equation] (cid:96) t ( X t , a ) (cid:3) .
Proof.
The proof is based on an application of the Fenchel–Young inequality, which requires theintroduction of the Legendre–Fenchel conjugate of g with respect to its ﬁrst argument.
This functionis deﬁned for all u ∈ R as
[Equation]
where the second equality and the inequality follow from a set of straightforward calculationsdeferred to Appendix A.4.
Turning to the actual proof, we consider the instantaneous pseudo-regretr t = (cid:96) ( θ (cid:63) X t , A t ) − (cid:96) ( θ (cid:63) , X t , A (cid:63)t ) in a ﬁxed round t and write the following (for any η > 0 ): E t [ r t ] = E t [ (cid:96) ( θ (cid:63) , X t , A t ) − (cid:96) ( θ (cid:63) , X t , A (cid:63)t )] =
E t [ (cid:96) ( θ
[Equation]
, X t , A t ) −
[Equation]
( θ t , X t , A t )] (using that ( θ (cid:63) , A (cid:63)t ) has the same conditional distribution as ( θ t , A t ) ) = E t [Equation]
¯
[Equation]
t ( X t , A t ) − (cid:96) ( θ t , X t , A t )
[Equation]
(by conditional independence of θ (cid:63) and A t )
[Equation]
(by the Fenchel-Young inequality)
[Equation]
+
I { A
[Equation]
= a }
[Equation]
(by Equation 5)
[Equation]
(by the tower rule of expectation and E t (cid:2)
[Equation]
(using again that θ t has the same conditional distribution as θ (cid:63)
[Equation]
Choosing the value of η > 0 for which the latter expression is minimal, we obtain E t [ r t ] ≤ 329 (cid:113) [-
-][Equation] t ( θ ∗ ;
[Equation] t ) (cid:80) a ∈A E t [Equation] ¯ (cid:96) t ( X t , a ) (cid:3) .
The proof is completed by taking the square on both sides andrearranging.
Conclusion
We have presented a new theoretical framework for analyzing Thompson sampling in contextualbandits, resulting in new results that advance the state of the art in the well-studied problem of logisticbandits.
We believe that these results are encouraging and that our analytic framework may ﬁnd manymore applications in the future.
As always, we leave many more questions open than what we have closed.
One major questionregarding logistic bandits is if it is possible to improve our new results by signiﬁcantly toning downthe dependence on the number of actions K .
In light of existing hardness results for nonlinear banditproblems (e.g., for generalized linear bandits with ReLU activation, Dong et al., 2021, Foster et al.
we suspect that this may not be possible.
As a more modest goal, we are curious to ﬁnd out
ifthe lifted information ratio can be upper bounded in terms of the smallest-slope parameter κ as donein many other works on logistic bandits since [Filippi et al., 2010].
We conjecture that a O ( κ − 2 d )bound on the lifted information ratio is indeed possible, but we were not able to prove it so far.
Thisis the case for the eluder dimension [Russo and Van Roy, 2013], another complexity measure that hasbeen used to upper-bound the regret for contextual bandits.
The eluder dimension for linear losses is
O ( d ) , but for nonlinear losses we know only of O ( κ − 2 d ) bounds for the generalized linear case.
More broadly, we believe that the most interesting immediate challenge is to extend our results tohold beyond the Bayesian setting.
As a counterexample by Zhang [2021] shows, this may not bepossible in general, but we wonder if his “feel-good” adjustment of Thompson sampling could beanalyzed with the techniques we introduced in this paper.
Throughout the paper we have studied several different settings, some of which come with a widerange of possible choices for the form of the prior and the likelihood.
In some of this scenarios,updating the posterior and sampling from it may be computationally challenging.
We have ignoredthis aspect in order to focus on the pure online-decision aspects and implicitly assumed that posteriorsampling can be performed without approximations. In practice, several heuristics have been proposed,see for instance [Dumitrascu et al., 2018] on efﬁcient TS for logistic bandits. It would be interestingto study how approximate sampling affects our regret guarantees, along the line of [Phan et al., 2019, Mazumdar et al., 2020].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] This is done in Section 6.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Ourresults are of the theoretical nature, so they don’t have any social impact.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are stated
in Section 5 and in the appendix of the paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]