Lifting the Information Ratio: An Information-Theoretic Analysis of Thompson Sampling for Contextual Bandits
Abstract
We study the Bayesian regret of the renowned Thompson Sampling algorithm in contextual bandits with binary losses and adversarially-selected contexts.
We adapt the information-theoretic perspective of Russo and Van Roy [2016] to the contextual setting by considering a lifted version of the information ratio deﬁned in terms of the unknown model parameter instead of the optimal action or optimal policy as done in previous works on the same setting.
This allows us to bound the regret in terms of the entropy of the prior distribution through a remarkably simple proof, and with no structural assumptions on the likelihood or the prior.
The extension to priors with inﬁnite entropy only requires a Lipschitz assumption on the log-likelihood.
An interesting special case is that of logistic bandits with d -dimensional parameters, K actions, and Lipschitz logits, for which we provide a (cid:101) O ( √ dKT ) regret upper-bound that does not depend on the smallest slope of the sigmoid link function.
Introduction
Thompson sampling is one of the most popular algorithms for sequential decision making under uncertainty.
First proposed by Thompson [1933], it has been rediscovered several times over the consequent decades and has been eventually popularized in the machine learning literature by
Chapelle and Li [2011] and Scott [2010], who pointed out its excellent empirical performance for solving contextual bandit problems.
These empirical studies were followed by a sequence of breakthroughs on the front of theoretical analysis, spearheaded by the works of Agrawal and Goyal [2012, 2013a,b], Kaufmann et al.
[2012] and Russo and Van Roy [2014], Russo and Van Roy [2016].
Thanks to these successes, Thompson sampling has become one of the gold-standard methods for solving multi-armed bandit problems.
Indeed, in the last decade, several Thompson-sampling-style methods have been developed and analyzed for a variety of problem settings.
The variety of different analysis techniques applied to Thompson sampling is perhaps even larger than the variety of problem settings that it has been applied to.
The ﬁrst key tools for analyzing the
Bayesian regret of Thompson sampling for multi-armed bandits have been developed by Russo and
Van Roy [2014], and our analysis naturally borrows several of these tools.
The worst-case results developed in said work were reﬁned in [Russo and Van Roy, 2016], where they proved for the ﬁrst time “information-theoretic” bounds on the regret of TS that scale with the Shannon entropy of the optimal action A ∗ under the prior on the model parameters.
This result has inspired a range of follow-up works, including extensions to uncountable action sets [Dong and Van Roy, 2018], approximate implementations [Lu and Van Roy, 2017, Qin et al., 2022], and even new algorithms based on the analysis technique itself [Russo and Van Roy, 2018, Kirschner and Krause, 2018].
One limitation that this technique could not overcome so far is not being able to satisfyingly deal with context .
When considering i.i.d.
contexts and ﬁnite policy classes, one can apply the theory of Russo and
Van Roy [2016] and treat policies as actions (as done by Liu et al.
2018, Bubeck and Sellke 2020), or adapt the information-ratio directly (as done by Kirschner et al.
2020, Hao et al. 2022) to obtain regret bounds scaling with the entropy of the optimal policy.
This approach, however, can lead to a polynomial dependence on the number of contexts when assuming no further structure about the rewards, as the entropy of the optimal policy may be as large as log | Π | = |X| log |A| .
Another variant of Thompson sampling demonstrating a similar prior-dependent regret bound has been proposed by Li [2013], whose regret guarantees also suffer from a suboptimal dependence on the number of rounds
T .
A much more satisfying solution has been recently given by Zhang [2021], whose “feel-good
Thompson sampling” method guarantees both frequentist and Bayesian regret bounds of the order √
KT log N , where K is the number of actions, T the number of rounds, and N the support size of the prior distribution on model parameters θ (cid:63) .
Under a Lipschitzness assumption on the prior and the likelihood, Zhang proves a frequentist regret bound scaling with the log-prior-probability mass assigned to the true parameter θ (cid:63) .
The techniques involved in proving these results drew substantial inspiration from the works of Foster and Rakhlin [2020], Foster and Krishnamurthy [2021] and in fact results of a similar ﬂavor were also proved recently in Foster et al.
Our own approach can be seen as a reconciliation of the analysis style of Zhang [2021] with the information-theoretic methodology of Russo and Van Roy [2016].
Our main conceptual contribution is proposing an adjustment of the now-classic notion of “information ratio” proposed by Russo and
Van Roy [2016] that applies to contextual bandits.
In its original deﬁnition, the information ratio quantiﬁes the tradeoff between incurring low regret and gaining information about the optimal action.
As we will argue, this notion of information gain is inappropriate for contextual bandits with noni.i.d.
contexts.
We propose a variant that measures the amount of information gained about the true model parameter θ (cid:63) instead of the optimal action (which may be context dependent).
The complexity notion resulting from this extension is called the “lifted information ratio”.
Our analysis shows that the Bayesian regret of Thompson sampling can be bounded in terms of the lifted information ratio and the Shannon entropy of the hidden parameter θ (cid:63) , which mirrors the result of Russo and Van Roy [2016].
Along the way, we draw inspiration from the recently proposed analysis technique of Zhang [2021] for contextual bandits, and in fact we show that our notion of lifted information ratio bridges the concept of “decoupling coefﬁcient” proposed by Zhang with the information ratio of Russo and
Van Roy.
We state our main results in the context of K -armed contextual bandits with binary losses.
For countable parameter spaces, we prove that the Bayesian regret of Thompson sampling satisﬁes a bound of order (cid:112) KTH ( θ ∗ ) , where H denotes the Shannon entropy.
This result is comparable to the bound of Foster and Krishnamurthy [2021] for the FastCB algorithm, which is of the order (cid:112)
KT log | Θ | and holds in a frequentist sense.
This is the best result we are aware of for this setting.
To demonstrate the ﬂexibility of our technique, we provide an extension to logistic bandits with
Lipschitz continuous logits, generalizing the well-studied setting involving logits that are linear functions of the context and the parameter θ .
For this setting, we prove a regret bound of order (cid:112)
[Equation]
) , where N ε (Θ , (cid:107)·(cid:107) ) is the ε -covering number of Θ under norm (cid:107)·(cid:107) and C is the Lipschitz constant of the logits.
This implies a regret bound of order √ KdT in the well-studied case of linear logits.
Notably, the bound does not show any dependence on the smallest slope κ of the sigmoid link function that almost all existing results for this setting suffer from [Filippi et al., 2010,
Kveton et al., 2020, Abeille et al., 2021, Faury et al., 2022].
Indeed, this constant has plagued all regret bounds since the early work of Filippi et al.
[2010] and was only recently moved to lower-order terms by the breakthrough work of Abeille et al.
Bounds involving other potentially large problem- dependent constants have also been proved in the Bayesian setting by Dong and Van Roy [2018] and
Dong et al.
To our knowledge, our bounds are the ﬁrst to entirely remove this factor.
The rest of the paper is organized as follows.
After introducing the necessary technical background in Section 2, we discuss matters of information gains, information ratios, and decoupling coefﬁcients in Section 3.
We state our main results and instantiate them in a variety of settings in Section 4.
We provide the key ideas of the analysis in Section 5 and conclude in Section 6.
Notation.
For a natural number n ∈ N , [ n ] = { 1 , 2 , . . . , n } denotes the set of the ﬁrst n natural numbers.
For x, y ∈ R d , (cid:104) x, y (cid:105) denotes the canonical scalar product of x and y , and (cid:107) x (cid:107) the Euclidean norm of x .
We denote the Shannon entropy of a discrete random variable Z with probability mass function P as H ( Z ) =
− E [log( P ( Z ))] .
We use ¯0 d for a d -dimensional vector of zeros and I d for the d × d identity matrix.
Preliminaries
We consider a parametric class of contextual bandits with parameter space Θ , context space X , and
K actions.
To each parameter θ ∈ Θ there corresponds a contextual bandit with loss distribution
P θ,x,a for each context x ∈ X and action a ∈ A , with the mean of the loss distribution denoted by (cid:96) ( θ, x, a ) .
We will dedicate special attention to the case where the losses are binary and thus P θ,x,a is a Bernoulli distribution with parameter (cid:96) ( θ, x, a ) .
For the main part of our theoretical analysis, we will assume Θ is either a ﬁnite set or a bounded metric space.
We study the problem of regret minimization in the Bayesian setting.
In this setting, the environment secretly samples a parameter θ (cid:63) from a known prior distribution Q 1 over Θ .
We assume that the agent has full knowledge of the prior and the likelihood model P θ,x,a .
The agent interacts with the environment for T rounds as follows.
At each round t ∈ [ T ] , an adaptive adversary selects a context X t , possibly using randomization and taking into account the previous history of actions and losses, but not θ (cid:63) . The latter is a common assumption [e.g., Agrawal and Goyal, 2013b].
After observing the context X t , the agent selects an action A t ∈ A (possibly using randomization) and incurs a binary loss L t ∼ P θ (cid:63) ,X t ,A t .
The goal of the agent is to minimize the expected sum of losses.
In the Bayesian setting, this is equivalent to minimizing the Bayesian regret, deﬁned as follows:
[Equation]
where A (cid:63)t = arg min a ∈A (cid:96) ( θ (cid:63) , X t , a ) is the optimal action for round t , and the expectation in (1) is over all sources of randomness: the initial sampling of θ (cid:63) from Q 1 , the agent’s randomization over actions, and the randomness of the loss realizations.
Furthermore, let F t = σ ( X 1 , A 1 , L 1 , . . .
, X t , A t , L t ) be the sigma-algebra representing the history of contexts, actions and losses observed by the agent up to time t included.
We use Q t to denote the distribution of the unknown parameter θ (cid:63) conditional on the past history F t − 1 , and simply call it the posterior distribution.
We denote by π t ( ·| X t ) the distribution over the agent’s actions conditional on
X t and F t − 1 , and call it the agent’s policy.
Finally, we will frequently use the shorthand notations E t [ · ] = E [ · |F t − 1 , X t ] and P t [ · ] = P [ · |F t − 1 , X t ] .
This paper is dedicated to the study of the celebrated Thompson Sampling (TS) algorithm, deﬁned as follows.
At each round t , TS draws a parameter θ t from the posterior distribution Q t .
Then, it selects the action that minimizes (cid:96) ( θ t , X t , · ) .
Finally, it updates Q t via Bayes’ rule, obtaining the new posterior Q t +1 .
The algorithm can be equivalently deﬁned as a method that plays actions according to their posterior probability of being optimal, that is: P [ A t = a |F t − 1 , X t ] =
P [ A (cid:63)t = a |F t − 1 , X t ] .
The pseudocode is shown as Algorithm 1.
Regret, information ratio, and decoupling coefﬁcient
The classic results of Russo and Van Roy [2016] establish that the regret of Thompson sampling in non-contextual multi-armed bandit problems can be upper bounded in terms of a quantity called the 1: Input: prior Q2: for t = 1 , 2 , . . , T do 3: Sample θ t ∼ Q t 4:
Observe context X t 5:
Play action A t = arg min a ∈A (cid:96) ( θ t , X t , a ) 6: Incur loss L t 7: Update Q t with ( X t , A t , L t ) to obtain Q t +1 by Bayes’ rule 8: end for information ratio . Informally, the information ratio measures the tradeoff between achieving low regret and gaining information about the identity of the optimal action A (cid:63) (which is a deterministic function of θ (cid:63) in the standard multi-armed bandit setting).
The formal deﬁnition is given by
[Equation]
where I t ( A (cid:63)t ; ( A t , L t )) denotes the mutual information between A (cid:63) and the action-observation pair ( L t , A t ) , conditioned on the history F t − 1 .
Intuitively, having small information ratio implies that every time Thompson sampling suffers large regret, it has to gain a lot of information about the optimal action, which suggests that it should be possible to bound the total regret by the total amount of information that there is to be gained.
The result of Russo and Van Roy [2016] conﬁrms this intuition by showing that the regret of Thompson sampling is of the order (cid:112) ρ ∗ TH
( A (cid:63) ) , where ρ ∗ = max t ρ ∗ t and H is the Shannon entropy.
The information ratio itself can always be upper bounded by K/ 2 , but better bounds can be shown when the loss function has favorable structural properties (e.g., whenever the reward function is a d -dimensional linear function, the information ratio is at most d/ 2 ).
While this result and the underlying information-theoretic framework is very elegant, it is inappropriate for studying contextual bandit problems.
The speciﬁc challenge is that the optimal action A (cid:63)t changes from round to round and gaining large amount of information about A (cid:63)t for any given round may not necessarily be useful for predicting future actions.
To see this, consider a stylized example with action set { 1 , 2 , . .
, K } , where there exists an action a reveal
= 1 whose loss entirely reveals the identity of the optimal action a (cid:63) ( x ) for context x : (cid:96) ( θ (cid:63) , x, a reveal )
= 1 − 10 − a (cid:63) ( x ) .
Taking this action provides large information gain about A (cid:63)t , but results in large regret and reveals nothing about the future losses.
Thus, in the contextual setting, one can keep following a policy that provides low information ratio while suffering linear regret.
This issue necessitates an alternative deﬁnition that still permits an effective information-theoretic analysis.
Our proposition is to consider a relaxed deﬁnition of the information ratio based on the mutual information between the true parameter θ (cid:63) and the observed loss.
In particular, we deﬁne
[Equation]
where I t ( θ (cid:63) ; L t ) =
E t (cid:2) D KL (cid:0)
P
L t | θ (cid:63) , F t − 1 ,X t ,A t (cid:13)(cid:13)
P L t |F t − 1 ,X t ,A t (cid:1)(cid:3) is the mutual information be-
tween θ (cid:63) and L t , conditioned on the history F t − 1 and the context-action pair X t , A t .
This quantity measures the information that the agent gains about θ (cid:63) .
High values of I t ( θ (cid:63) ; L t ) intuitively allow making better predictions about the future loss realizations for all possible context sequences.
Since
A (cid:63)t is a deterministic function of θ (cid:63) given X t , the data processing inequality implies that the information gain about θ (cid:63) is always greater than that about A (cid:63)t , which in turn implies that ρ t is smaller than what one would obtain by directly generalizing the deﬁnition of Russo and Van Roy [2016].
As this notion of information gain measures the efﬁciency of inferring the identity of a hidden parameter, we refer to I t ( θ (cid:63) ; L t ) as the lifted information gain , and ρ t as deﬁned in Equation (2) as the lifted information ratio.
As our analysis will establish, a bounded lifted information ratio guarantees low regret, and we will show that the ratio itself can be bounded reasonably under conditions similar to the ones required by the analysis of Russo and Van Roy [2016], despite the fact that all quantities are deﬁned conditionally on a sequence of non-i.i.d.
contexts..
We are not the ﬁrst to consider the lifted information ratio: very similar quantities have been considered in the literature on information-directed sampling , as initiated by Russo and Van Roy
[2018], who have already introduced an the non-contextual counterpart of the same quantity in their Section 9.2.
The recent works of Kirschner et al.
[2020], Hao et al.
[2022] also consider versions of the standard information ratio for contextual bandits, but their analysis crucially relies on the assumption that the contexts are i.i.d.
Similarly, Lu et al.
[2021], Liu et al.
[2022] deﬁne versions of the information ratio using various targets for information-directed sampling, which recovers our deﬁnition of the lifted information ratio when setting the target as θ (cid:63) .
While these works do not distinguish between all these versions of the information ratio on the level of terminology, we believe that emphasizing the “lifted” nature of our information ratio provides more clarity in that it indicates the speciﬁc features of contextual bandit learning with parametric losses in a transparent way.
Thus, we will keep using this terminology throughout the paper.
Our lifted information ratio is also closely related to a quantity appearing in the analysis of Zhang [2021], called the “decoupling coefﬁcient”.
Adapted to our Bayesian setting, this coefﬁcient can be deﬁned as the smallest constant δ such that the following inequality holds:
[Equation]
where the ﬁrst line gives the original deﬁnition mirroring that of Zhang [2021] and the second line plugs in the choice of µ achieving the inﬁmum.
Reordering gives the value of the optimal δ ∗ t :
[Equation]
which matches our deﬁnition of the lifted information ratio, up to the difference of replacing the mutual information by the root mean-squared error in predicting the true parameter θ ∗ .
Notably, this deﬁnition essentially coincides with the lifted information ratio for the special case of Gaussian losses.
Main results
In this section, we state our main results concerning the Bayesian regret of Thompson sampling for contextual bandits.
We will assume that the losses are binary and the action space is ﬁnite, unless otherwise stated.
However, several of our results can be generalized beyond this setting.
We will illustrate this in Section 4.3, where we provide some additional results for the classic setting of
Gaussian linear contextual bandits.
We begin by stating two general regret bounds in terms of the lifted information ratio deﬁned in
Equation (2).
The reader that is not interested in the full generality of our theory may skip to
Section 4.2 for concrete regret bounds.
Our ﬁrst abstract bound applies to priors with ﬁnite entropy, the simplest example being ﬁnite parameter spaces.
Theorem 1.
Assume Q 1 is supported on the countable set Θ 1 ⊆ Θ and that the lifted information ratio for all rounds t satisﬁes ρ t ≤ ρ for some ρ > 0 .
Then, the Bayesian regret of TS after T rounds can be bounded as
[Equation]
In particular if Θ 1 is a ﬁnite set with | Θ 1 | =
N , the regret of TS satisﬁes
[Equation]
The proof of this theorem is stated in Section 5.1.
Unfortunately, the Shannon entropy can be unbounded for distributions with inﬁnite support, which is in fact the typical situation that one encouters in practice.
To address this concern, we develop a more general result, that holds for a broader family of distributions.
In the following, (Θ , (cid:37) ) is a metric space with metric (cid:37) :
Θ 2 → R .
We make the following regularity assumption on the likelihood function P θ,x,a :
Assumption 1.
There exists a constant C > 0 such that for any θ, θ (cid:48)
∈ Θ 1 , | log P θ,x,a ( L ) − log P θ (cid:48) ,x,a ( L ) | ≤ C(cid:37)
( θ, θ (cid:48) ) holds for all x ∈ X , a ∈ A , and L ∈ { 0 , 1 } .
Under this assumption, we can state a variant of Theorem 1 that applies to metric parameter spaces:
Theorem 2.
Assume (Θ , (cid:37) ) is a metric space, and Q 1 is supported on Θ 1 ⊆ Θ with ε -covering number N ε (Θ 1 , (cid:37) ) .
Let Assumption 1 hold, and assume the lifted information ratio for all rounds t satisﬁes ρ t ≤ ρ for some ρ > 0 .
Then, the Bayesian regret of TS after T rounds can be bounded as
[Equation]
The proof is based on a covering argument on top of the proof of the previous theorem, and is provided in Appendix A.2.
To get a better understanding of Assumption 1, it is useful to notice that it is satisﬁed in basic settings, like logistic bandits with Lipschitz logits.
See Section 4.2 for details.
4.1 Bounding the lifted information ratio
At this point, some readers may worry that the lifted information ratio may be impossible to bound due to the lifting to the space of parameters Θ .
To address this concern, we now turn to showing bounds on the lifted information ratio.
We ﬁrst consider the unstructured case, that holds for arbitrary parameter spaces, likelihoods, and priors.
Lemma 1.
Suppose that the losses are binary and |A| =
K .
Then, the lifted information ratio of
Thompson sampling satisﬁes ρ t ≤ 2 K for all t ≥ 1 .
The proof of this lemma (provided in Section 5.2) relies on a decoupling argument between the choice of the action and of the parameters at round t , inspired by Zhang [2021].
Taking his argument one step further, we center our analysis around an application of convex conjugacy, which we believe may be applicable in a broader variety of settings.
We wish to highlight that this proof technique is very different from the information-theoretic methodology pioneered by Russo and Van Roy [2016].
Next, we consider the case of linear expected losses in Euclidean parameter spaces, which, in principle, allows for an unbounded number of actions.
Lemma 2.
Suppose that Θ ⊆ R d , the losses are binary, and the expected losses are linear functions of the form (cid:96) ( θ, x, a ) = (cid:104) θ, ϕ ( x, a ) (cid:105) , where ϕ :
X × A → R d is a feature map, such that (cid:96) ( θ, x, a ) ∈ [0 , 1] for all x ∈ X , a ∈ A , θ ∈ Θ .
Then, the lifted information ratio satisﬁes ρ t ≤ d/ 2 .
The proof of this result (deferred to Appendix B) follows the arguments of Proposition 5 of Russo and Van Roy [2016], with some small but important changes to account for the presence of contexts.
Notably, both of these results match the classic bounds of Russo and Van Roy [2016] on the standard deﬁnition of the information ratio for these settings (cf.
their Propositions 5 and 3), implying that lifting to the space of parameters does not substantially impact the regret-information tradeoff.
4.2 Concrete regret bounds for Bernoulli bandits
We now instantiate our bounds in two well-studied settings for Bernoulli bandits.
We start from the fully unstructured case, assuming ﬁnite actions and ﬁnitely supported prior.
The following regret bound follows directly from Theorem 1 and Lemma 1.
Theorem 3.
Consider a contextual bandit with K actions and binary losses, and suppose Θ 1 , the support of Q 1 , is ﬁnite with | Θ 1 | = N .
Then, the Bayesian regret of TS satisﬁes:
[Equation]
This result is comparable to the best known regret guarantees for this problem due to Foster and
Krishnamurthy [2021] and Zhang [2021], and matches the minimax rate for unstructured contextual bandits with a policy class of size N [Beygelzimer et al., 2011, Dudík et al., 2011].
However, we are not aware of a comparable result for the Thompson sampling algorithm in the literature, be it
Bayesian or not.
Moving to Bernoulli bandits with structure, we consider a well-studied setting known as logistic bandits .
In this model, the losses are generated by a Bernoulli distribution as L t ( θ, x, a ) ∼
Ber( σ ( f θ ( x, a ))) , where σ ( z ) =
1 / (1 + e − z ) is the sigmoid function.
We just assume that f θ ( x, a ) , called the logit function, is C -Lipschitz in θ , which directly implies that Assumption 1 holds.
Notice that our deﬁnition generalizes the commonly used notion of logistic bandits that consider linear logit functions of the form f θ ( x, a ) = (cid:104) θ, ϕ ( x, a ) (cid:105) , where ϕ :
X × A → R d is some feature map.
Our result for logistic bandits is based on Theorem 2 and Lemma 1.
Theorem 4.
Assume Θ ⊂ R d and (cid:107) θ (cid:107) ≤ S for all θ ∈ Θ 1 .
Consider a class of logistic bandits with
K actions and C -Lipschitz logit function f θ ( x, a ) .
Then, the Bayesian regret of TS after T rounds can be bounded as:
[Equation]
The proof follows from an application of Theorem 2, and can be found in Appendix A.3.
To our knowledge, this is the ﬁrst regret bound for logistic bandits with nonlinear logits.
We can further specialize our result to linear logits, the setting that is most commonly studied in the literature:
Corollary 1.
Assume Θ ⊂ R d , (cid:107) θ (cid:107) ≤ S for all θ ∈ Θ 1 .
Consider a class of logistic bandits with
K actions and linear logit function f θ ( x, a ) = (cid:104) ϕ
( x, a ) , θ (cid:105) , with (cid:107) φ ( x, a ) (cid:107) ≤ B for all x ∈ X and a ∈ [ K ] .
Then, the Bayesian regret of TS after T rounds can be bounded as:
[Equation]
The proof of the corollary is stated at Appendix A.3.
A remarkable feature of this bound is that it shows no dependence on the minimum derivative of the sigmoid link function, albeit at the price of a √ K factor in the bound.
Nevertheless, we believe this to be the ﬁrst regret guarantee that entirely gets rid of this potentially enormous constant without very strong assumptions.
Indeed, this constant has been present in nearly all previous bounds we are aware of [Filippi et al., 2010, Li et al., 2017,
Faury et al., 2020, Abeille et al., 2021, Faury et al. , 2022]—although these results have the advantage of holding in a frequentist sense.
In the Bayesian setting, the works of Dong and Van Roy [2018] and Dong et al.
[2019] have proved a variety of bounds on the regret of Thompson sampling for non-contextual logistic bandits, but none of them are directly comparable with our result above.
Dong et al.
[2019] prove a regret bound of order d √ T for a highly specialized setting with A = Θ , and a range of other bounds under a variety of strong assumptions.
An improved bound for the many-actions setting that scales at most logarithmically with K remains an open problem.
Its difﬁculty is testiﬁed by a set of negative examples provided by Dong et al.
[2019], and by a long-lived conjecture of [Dong and Van Roy, 2018] regarding the information ratio for logistic bandits, that, to our knowledge, has not yet been veriﬁed in theory.
4.3 Beyond binary losses: linear bandits with Gaussian noise
We now illustrate how our techniques (in particular the lifted information ratio) can be extended beyond the case of binary losses, and in particular consider the classic setting of Bayesian linear contextual bandits, where the loss is a linear function of a d -dimensional feature map with additive
Gaussian noise, and the prior is also Gaussian.
Lemma 3.
Suppose θ ∗ be a d -dimensional normal random vector, so θ ∗ ∼ N (¯0 d , λI d ) for some λ > 0 , the loss is a normal random variable L t ∼ N ( (cid:96) ( θ (cid:63) , X t , A t ) , σ 2 ) for some standard deviation σ > 0 , and the expected loss is linear, so (cid:96) ( θ, x, a ) =
(cid:104) θ,
ϕ ( x, a ) (cid:105) , where ϕ : X ×
[ K ] → R d is a feature map.
Then, the lifted information ratio satisﬁes ρ t ≤ min { d, 2(1 + log K ) } .
For this setting, the bound ρ t ≤ d has already been shown for the classic information ratio by
Russo and Van Roy [2016].
However, we believe that our bound ρ t ≤ 2(1 + log K ) is new for any deﬁnition of information ratio.
By combining this bound on the lifted information ratio with standard arguments for linear contextual bandits, we recover both of the well-known regret bounds of order d √ T log T and (cid:112) dT log ( KT ) for this seting, respectively due to Abbasi-Yadkori et al.
[2011] and Chu et al.
2 See Corollary 2 in Appendix B for a rigorous statement and the proof.
This result, although not surprising, indicates once again that our notion of lifted information ratio does not lead to compromises in performance, even when the losses are not binary.
Analysis
This section presents the key ideas of the proofs of our main results.
We will just provide the proof of
Theorem 1 and that of Lemma 1, which we believe offer the most insight into our techniques.
All other proofs, included those of auxiliary lemmas, are deferred to Appendix A and B. For sake of clarity, we focus on the relatively simple case where Θ 1 is countable, so that (with a slight abuse of notation) we can write Q t ( θ ) to denote the posterior probability associated with θ .
Note, however, that our full proofs also handle the case of general distributions (details in Appendix A.1).
5.1 The proof of Theorem
Recalling the deﬁnition of the lifted information ratio (Equation 2), we ﬁrst notice that the regret can be rewritten as follows:
[Equation]
where the ﬁrst step uses the tower rule of expectation, the second step the deﬁnition of ρ t , and the ﬁnal step follows from the Cauchy–Schwarz inequality.
The key challenge is then to bound the sum of information-gain terms.
The following lemma provides a more tractable form of this sum:
Lemma 4.
Under the assumptions of Theorem 1,
[Equation]
The proof of this lemma is based on a classic “Bayesian telescoping” argument that we have learned from Grünwald [2012].
We provide the proof of Lemma 4 in Appendix A.1.
Supposing now that the prior has bounded entropy, we can easily bound the term appearing on the right hand side as follows:
[Equation]
This concludes the proof of the ﬁrst statement.
The second statement follows from the ﬁrst using the trivial bound on the Shannon entropy of any ﬁnite-support distribution.
5.2 The proof of Lemma
We start by introducing some notation that will be useful for the proof.
In particular, we use g to denote the binary relative entropy function deﬁned for all p, q ∈ [0 , 1] as
[Equation]
(4) and we use the convention 0 · log 0 = 0 .
Furthermore, we deﬁne the posterior mean loss as (cid:96) t [Equation] =
[Equation] t [ [Equation] ( θ (cid:63) , x, a )] .
These notations allow us to conveniently rewrite the information gain as
I t ( θ (cid:63) ; L t ) = E t (cid:2) D KL (cid:0)
P
L t | θ (cid:63) , F t − 1 ,X t ,A t (cid:13)(cid:13)
P L t |F t − 1 ,X t ,A t (cid:1)(cid:3) =
E t (cid:2) g (cid:0) (cid:96) ( θ (cid:63) , X t , A t ) (cid:13)(cid:13) (cid:96) t
( X t , A t ) (cid:1)(cid:3) .
We will now prove a generalization of Lemma 1, which will directly imply the original result:
Lemma 5.
Under the assumptions of Lemma 1, for all t , the lifted information ratio of Thompson sampling satisﬁes
[Equation]
Proof.
The proof is based on an application of the Fenchel–Young inequality, which requires the introduction of the Legendre–Fenchel conjugate of g with respect to its ﬁrst argument.
This function is deﬁned for all u ∈ R as
[Equation]
where the second equality and the inequality follow from a set of straightforward calculations deferred to Appendix A.4.
Turning to the actual proof, we consider the instantaneous pseudo-regret r t = (cid:96) ( θ (cid:63) X t , A t ) − (cid:96) ( θ (cid:63) , X t , A (cid:63)t ) in a ﬁxed round t and write the following (for any η > 0 ):
E
[Equation]
t )
[Equation]
(using that ( θ (cid:63) , A (cid:63)t ) has the same conditional distribution as ( θ t , A t ) ) =
[Equation]
( X t , A t )
[Equation]
( θ t , X t , A t ) (cid:3) (by conditional independence of θ (cid:63) and A t )
[Equation]
(by the Fenchel-Young inequality)
[Equation]
Equation 5) [Equation]
(by the tower rule of expectation and E t (cid:2) I
[Equation]
A t = a }
[Equation]
π t ( a | X t )
[Equation]
(using again that θ t has the same conditional distribution as θ (cid:63) )
[Equation]
Choosing the value of η > 0 for which the latter expression is minimal, we obtain E t [ r t ] ≤ (cid:113) {+
+}
I t ( θ ∗ ; L t ) (cid:80) a ∈A E t (cid:2) ¯ (cid:96) t ( X t , a ) (cid:3) .
The proof is completed by taking the square on both sides and rearranging.
Conclusion
We have presented a new theoretical framework for analyzing Thompson sampling in contextual bandits, resulting in new results that advance the state of the art in the well-studied problem of logistic bandits.
We believe that these results are encouraging and that our analytic framework may ﬁnd many more applications in the future.
As always, we leave many more questions open than what we have closed.
One major question regarding logistic bandits is if it is possible to improve our new results by signiﬁcantly toning down the dependence on the number of actions K .
In light of existing hardness results for nonlinear bandit problems (e.g., for generalized linear bandits with ReLU activation, Dong et al., 2021, Foster et al., 2021) we suspect that this may not be possible.
As a more modest goal, we are curious to ﬁnd out if the lifted information ratio can be upper bounded in terms of the smallest-slope parameter κ as done in many other works on logistic bandits since [Filippi et al., 2010].
We conjecture that a O ( κ − 2 d ) bound on the lifted information ratio is indeed possible, but we were not able to prove it so far.
This is the case for the eluder dimension [Russo and Van Roy, 2013], another complexity measure that has been used to upper-bound the regret for contextual bandits.
The eluder dimension for linear losses is
O ( d ) , but for nonlinear losses we know only of O ( κ − 2 d ) bounds for the generalized linear case.
More broadly, we believe that the most interesting immediate challenge is to extend our results to hold beyond the Bayesian setting.
As a counterexample by Zhang [2021] shows, this may not be possible in general, but we wonder if his “feel-good” adjustment of Thompson sampling could be analyzed with the techniques we introduced in this paper.
Throughout the paper, we have studied several different settings, some of which come with a wide range of possible choices for the form of the prior and the likelihood.
In some of this scenarios, updating the posterior and sampling from it may be computationally challenging.
We have ignored this aspect in order to focus on the pure online-decision aspects, and implicitly assumed that posterior sampling can be performed without approximations.
In practice, several heuristics have been proposed, see for instance [Dumitrascu et al., 2018] on efﬁcient TS for logistic bandits.
It would be interesting to study how approximate sampling affects our regret guarantees, along the line of [Phan et al., 2019,
Mazumdar et al., 2020].
Acknowledgments and Disclosure of Funding
This project has received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (Grant agreement No. 950180).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] This is done in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our results are of the theoretical nature, so they don’t have any social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are stated in Section 5 and in the appendix of the paper.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
For didactic purposes, we provide two proofs for this lemma. We ﬁrst start with the simple case of distributions with ﬁnite supports that allows us to spell out the steps in the proof using simple and intuitive notation. Then, we provide a proof for general prior distributions.
Some general notations that we will use throughout are the following.
We let D θ,x,a be the distribution of the loss given context x , action a and a ﬁxed parameter θ ∈ Θ , and let L θ,x,a denote the random variable with said distribution. Using this notation, notice that L t ∼ D θ ∗ ,X t ,A t . Finally, we deﬁne the likelihood function p θ,t ( c ) = P
[ L θ,X t ,A t = c | X t , A t ]
Proof for countably supported priors.
We ﬁrst assume that the support Θ 1 ⊆
Θ is countable, which will allow us to reason about probability mass functions.
In particular, with a slight abuse of our notation, we will write Q t ( θ ) =
P t [ θ (cid:63) =
θ ] (which should otherwise be written as Q t ( { θ } ) ).
Deﬁning the Bayesian posterior predictive distribution p t ( c ) = (cid:80) θ Q t ( θ ) p θ,t ( c ) , we can write
[Equation]
Then, summing up and taking marginal expectations, we get
[Equation]
To proceed
[Equation]
let us notice that the posterior updates take the following form by deﬁnition:
[Equation]
Also
[Equation]
let us deﬁne the notation p
( L 1: T ) =
[Equation]
θ Q
[Equation]
θ ) (cid:81)
Tt =1 p θ,t ( L t ) and notice that we can express this quantity by a recursive application of the above expression as
[Equation]
Then, we have log
[Equation]
Proof for general prior distributions.
The proof follows from similar arguments, although we cannot work with probability mass functions any more.
In particular, we will denote by Q 1 the prior distribution of θ ∗ , which satisﬁes the following identity:
[Equation]
Similarly, we denote by Q t +1 the posterior distribution on θ ∗ after round t , which satisﬁes
[Equation]
We now apply Bayes theorem for general distributions that gives the following expression for q t +1 :
[Equation]
where q t +1 ( θ ) =
dQ t +1 dQ 1 ( θ ) is the Radon-Nykodim derivative of the posterior measure with respect to the prior measure which is always guaranteed to exist (cf.
Theorem 1.31 of Schervish, 1996).
As in the previous proof, we once again deﬁne p t ( L t ) =
(cid:82) θ
∈ Θ p θ,t ( L t ) q t ( θ ) dQ 1 ( θ ) and p ( L 1: T ) =
(cid:82) θ ∈ Θ (cid:81)
Tt =1 p θ,t ( L t ) dQ 1 ( θ ) , and compute the relation : p ( L 1: T ) =
[Equation]
t = p (
[Equation]
[Equation]
Then, we have log
[Equation]
Taking expectations and repeating the derivations of Equation (6) and Equation (7), the proof is concluded.
A.2 Proof of Theorem
The proof follows the same arguments as the proof of Theorem 1, except we need a different bound on the sum of information-gain terms that does not involve the entropy of Q 1 .
The bound is based on a covering argument that is provided by the following lemma, which, together with Equation (3), will directly imply the result claimed in the theorem.
Lemma 6.
Under the assumptions of Theorem 2, E
[Equation]
Proof.
For the clarity of exposition, we still assume that the support Θ 1 of Q 1 is a countable subset of Θ , even though our results can be extended to general distributions.
Fix an ε > 0 and let (cid:98) Θ ε ⊂ Θ be a minimal ε -cover of Θ 1 .
Thus, for any θ ∈ Θ 1 , there exists (cid:98) θ ∈ (cid:98) Θ ε such that (cid:37) ( θ, (cid:98) θ ) ≤ ε .
Let G ε be the partition of Θ 1 constructed by associating to each θ ∈ Θ 1 the closest (as measured by metric (cid:37) ) element of the cover (cid:98) θ ∈ (cid:98) Θ ε .
For any θ ∈ Θ 0 , we denote by G ε ( θ ) the unique set of the partition containing θ .
Note that, for any θ ∈ Θ 1 and (cid:101) θ ∈ G ε ( θ ) , ρ ( θ, (cid:101) θ ) ≤ 2 ε
and so, by Assumption 1, | log p θ,t ( L t ) − log p (cid:101) θ,t ( L t ) | ≤ 2 Cε .
Hence, we have − log (cid:32) (cid:88) θ ∈
Θq 1 ( θ )
T (cid:89) t = p θ,t ( L t ) (cid:33) ≤ − log   (cid:88) θ ∈G ε ( θ (cid:63) )
q 1 ( θ )
T (cid:89) t = p θ,t ( L t ) 
 = − log   (cid:88) θ ∈G ε ( θ (cid:63) )
q 1 ( θ )
T (cid:89) t = (cid:18) p θ (cid:63) ,t ( L t ) · p θ,t ( L t ) p θ (cid:63) ,t ( L t ) (cid:19) ≤ − log   (cid:88)
θ ∈G ε ( θ (cid:63) )
q 1 ( θ )
T (cid:89) t
= (cid:0) p θ (cid:63) ,t ( L t ) ·
e − 2 Cε (cid:1) 
 ≤ − log   (cid:88) θ ∈G ε ( θ (cid:63) )
q 1 ( θ )
T (cid:89) t =
p θ (cid:63) ,t ( L t ) 
 + 2 CεT
[Equation]
where we denoted (cid:98) Q 1 ( G
[Equation]
( θ (cid:63) )) = (cid:80) θ ∈G ε ( θ (cid:63) )
q 1 ( θ ) .
Using this result together with Lemma 4, we get
E
[Equation]
The proof of the lemma is concluded by noting that |G ε | ≤ | (cid:98) Θ (cid:15) | ≤ N ε (Θ 1 , (cid:37) ) .
A.
3 The proofs for logistic bandits
Proof of Theorem 4 Using a standard result on the covering number of the Euclidean ball, we have N ε (Θ 0 , (cid:107)·(cid:107) ) ≤ ( 2 Sε + 1) d .
Regarding Assumption 1, notice that log( σ ( f θ ( x, a )))
=
f θ ( x, a ) − log(1 + exp( f θ ( x, a ))) .
Since log( σ ( z )) and log(1 − σ ( z )) are 1 -Lipschitz and f θ ( x, a ) is CLipschitz, log P θ,x,a (1)
= log( σ ( f θ ( x, a )))
and log P θ,x,a (0)
= log(1 − σ ( f θ ( x, a ))) are also
C -Lipschitz, implying that Assumption 1 holds with the same constant C .
The claim then follows from Theorem 2 by taking ε = 1 /CT .
{+Proof of Corollary 
+}
The function f θ,x,a : θ → (cid:104) ϕ
( x, a ) , θ (cid:105) is linear in θ and since ϕ ( x, a ) is bounded in norm by B, it is also B -Lipschitz.
This proves that Assumption 1 is satisﬁed with the constant B .
We can then apply Theorem 4 with C = B and ρ = 2 K (Lemma 1).
A
4 The Legendre–Fenchel conjugate of the binary relative entropy
Proposition 1.
For any u ≤ 0 and q ∈ [0 , 1] :
[Equation]
Proof.
The claim follows from the following calculation:
[Equation]
where the ﬁrst inequality is from log(1 + x ) ≤ x for any x > − 1 , and the second inequality is from e x ≤ 1 + x + x 2 2 for any x ≤ 0 .
B Linear bandits
By making an assumption that loss function is a linear function of some feature map over states and actions, this linear dependence allows the learner to generalize the observation among the other actions.
We provide two different results for the linear bandits setting, the ﬁrst one is Lemmathat holds for general distributions with expected loss being a linear function taking values in (0 , 1) , and the other is Lemma 3 that works for setting where the prior of θ ∗ is Gaussian distribution and the observations have a Gaussian noise.
These two results are connected and share a similar proof techniques, thus we ﬁrst state the following auxiliary lemma, that holds for general linear functions.
We also need to deﬁne the following quantity:
[Equation]
We prove the following upper bound on this quantity:
Lemma 7.
Suppose the expected losses are linear functions of the form (cid:96) ( θ, x, a ) = (cid:104) θ, ϕ ( x, a ) (cid:105) , for all x ∈ X , a ∈ A , θ ∈ R d , where ϕ : X × A → R d is a feature map.
Then, δ t ≤ d .
Proof.
The proof follows from an adaptation of Proposition 5 of Russo and Van Roy [2016].
Their technique is based on constructing a K × K matrix M and showing that the information ratio can be upper bounded by the rank of this matrix.
In particular, let us ﬁx t and deﬁne λ a = π t ( a | X t ) and the matrix M whose elements are deﬁned as
[Equation]
First, we relate the denominator of δ t to this matrix as follows:
[Equation]
where we used Jensen’s inequality.
For the numerator
[Equation]
we get
[Equation]
Thus, we have (cid:0) E t (cid:2) ¯ (cid:96) t ( X t , A t ) − (cid:96) ( θ ∗ , X t , A ∗ t ) (cid:3)(cid:1) 2 = (cid:0) tr ( M ) (cid:1) 2 .
Putting this together with the previous bound, we conclude that ρ t ≤ ( tr ( M ) / (cid:107) M (cid:107) F ) 2 , which can be further upper bounded by the rank of M .
Now, one can follow the steps of the proof of Proposition 5 of Russo and Van Roy [2016] to prove that the rank of M is at most d .
Equipped with result of Lemma 7, we can derive our analysis for the two described linear settings:
The proof of Lemma 2 We relate the denominator of δ t , deﬁned in (9), to the denominator of ρ t , by using Pinsker’s inequality as follows:
[Equation]
Using the deﬁnition of ρ t and Lemma 7, we get that ρ t ≤ d 2 .
The proof of
Lemma 3 Note that for the
Gaussian likelihood,
I t ( θ ∗ ; L t ) =
E t (cid:104)(cid:0) ¯ (cid:96) t ( X t , A t ) − (cid:96) ( θ ∗ , X t , A t ) (cid:1) 2 (cid:105) , so we have
[Equation]
[Table]
Lemma 7 provides ρ t = δ t ≤ d bound.
We still can tighten this results for the cases when log K (cid:28) d by using the linearity and the closed form expression for I t ( θ ∗ ; L t ) .
Under the stated assumptions, the posterior Q t is N ( θ, Σ t ) with θ t = Σ − 1 t (cid:80) s<t ϕ ( X s , A s ) ϕ
( X s , A s ) T L s .
and Σ t = (cid:80) s<
t ϕ
[Equation]
X s , A s ) ϕ
{+( X s , A s ) T + σ 
+}
λ
I .
Note that ¯ (cid:96)
[Equation]
We ﬁrst consider the numerator of ρ t .
Let us deﬁne σ 2 t,a = ϕ ( X t , a ) T Σ − 1 t ϕ ( X t , a ) , and notice that conditional on F t , X t , we have (cid:104) θ t − ˆ θ t , ϕ
( X t , a ) (cid:105) σ − 1 t,a ∼ N (0 , 1) .
Let Z denote a standard normal, then for c > 0 , P [ Z > c ] ≤ 0 .
5 e − c 2 / 2 .
Using c δ = (cid:112) 2 log( K/δ ) , we can show
[Equation]
for all
[Equation]
Then, with probability at least 1 − δ ,
[Equation]
Note that:
[Equation]
In order to get the bound on this, we will use the idea of proof of Theorem 3.3 from Bubeck and
Cesa-Bianchi [2012]: taking W t =
( (cid:96) t ( X t ,A t ) − (cid:96) ( θ t ,X t ,A t )) 2 2 σ 2 t,At − ln K and using the formula
[Equation]
we get E t [ W ] ≤ 1 , which gives
[Equation]
Now, we notice that the denominator can be written as
[Equation]
Therefore, we can bound ρ t for all t as follows:
[Equation]
For the complete of our analysis, we also prove a regret bound for the Gaussian setting:
Corollary 2.
Consider the classic setting of linear bandits, such that the loss of playing action a at time t satisﬁes (cid:96) ( θ ∗ , x, a ) ∼ N ( (cid:104) θ (cid:63) , ϕ ( x, a ) (cid:105) , σ 2 ) , where θ ∗ is drawn from a Gaussian prior
Q 1 = N (0 , λI ) with some λ > 0 , σ > 0 and ϕ : X ×
[ K ] → R d is a feature map such that (cid:107) ϕ
( x, a ) (cid:107) ≤ B for all x ∈ X , a ∈ [ K ] .
Then, the Bayesian regret of TS satisﬁes
[Equation]
Proof.
By using the form of I t ( θ ∗ ; L t ) for the Gaussian likelihood and applying the regret decomposition of Equation (3), we get
[Equation]
Then, by Lemma 3 and (10),
[Equation]
The term (cid:80) Tt =1 E t
[Equation]
ϕ ( X t , A t ) T Σ − 1 t ϕ ( X t , A t ) (cid:3) can be addressed by using the elliptical potential lemma (e.g. Lemma 19.4.
in Lattimore and Szepesvári [2020]), which is widely used in the analysis of linear bandits.
Applying this lemma, we get
[Equation]
which completes the proof.