Residual Bootstrap Exploration for Stochastic Linear Bandit
Abstract
We propose a new bootstrap-based online algorithm for stochastic linear bandit problems.
The key idea is to adopt residual bootstrap exploration, in which the agent estimates the next step reward by re-sampling the residuals of mean reward estimate.
Our algorithm, residual bootstrap exploration for stochastic linear bandit ( LinReBoot ), estimates the linear reward from its re-sampling distribution and pulls the arm with the highest reward estimate.
In particular, we contribute a theoretical framework to demystify residual bootstrapbased exploration mechanisms in stochastic linear bandit problems.
The key insight is that the strength of bootstrap exploration is based on collaborated optimism between the onlinelearned model and the re-sampling distribution of residuals.
Such observation enables us to show that the proposed LinReBoot secure a high-probability ˜ O ( d Ô n ) sub-linear regret under mild conditions.
Our experiments support the easy generalizability of the ReBoot principle in the various formulations of linear bandit problems and show the signiﬁcant computational e  ciency of LinReBoot .
INTRODUCTION
Stochastic linear bandit is an online learning problem that the learning agent acts by pulling arms, where each arm is associated with a feature vector, then learning the arms information from the corresponding random rewards.
In such problems, the typical goal of a learning agent is to maximize its cumulative reward.
Learning more about an arm (explore) or pulling the arm with the highest estimated reward (exploit) leads to the well-known exploration- exploitation tradeo  , which is the central trade-o  captured in many decision-making applications in modern online service industries.
Consequently, the design of stochastic linear bandit algorithms demands an easy-generalizable implementation across various contextualize actions and reward generation processes.
In the past decade of bandit literature, such demands have invited researchers to investigate bootstrap-based exploration-exploitation trade-o  s and have drawn rising attention [Baransi et al., 2014, Eckles and Kaptein, 2014, Osband and Van Roy, 2015, Vaswani et al., 2018,
Hao et al., 2019, Kveton et al., 2019b, Wang et al., 2020].
Yet, prior works on bootstrap-based bandit algorithms focus on provable multi-armed bandit algorithms and only provide a limited empirical evaluation of bootstrap-based stochastic linear bandit algorithms, and their theoretical counterpart remains unknown.
Such knowledge gap of bootstrapping stochastic linear bandit persuades our investigation on the provable bootstrap-based stochastic linear bandits: Can we theoretically and empirically support the validity and easy-generalizability of bootstrapping procedure in stochastic linear bandit algorithms design?
In particular, we aim to deliver a generic framework to demystify the bootstrap optimism in stochastic linear bandit problems and validate the easy generalizability of the bootstrap principle across various contextual linear bandit problems.
Contributions.
We introduce LinReBoot algorithms that implement Residual Bootstrap Exploration for stochastic linear bandit problem with sub-linear regret.
We theoretically show that LinReBoot secures ˜ O
( d Ô n ) regret where d is the dimension of features.
This sublinear regret bound matches the regret bound of the same order as those theoretical results of Linear Thompson Sampling algorithms.
The key to achieving such sub-linear regret guarantee is to carefully manage and collaborate sample and bootstrap optimism (Section
In particular, by measuring the ”sample-bootstrap optimistic estimated discrepancy ratio” of the optimal arm, LinReboot successfully avoids over or under exploration and theoretically secures sub-linear mean regret with high-probability.
To our knowledge, this is the ﬁrst theoretical analysis to support the validity and e  ciency of the residual bootstrap-based procedure for stochastic linear bandit problems.
We empirically show that LinReBoot rivals or exceeds competing algorithms including Linear Thompson Sampling, Linear
PHE, Linear GIRO, and Linear UCB under stochastic linear bandit problem as well as more complicated linear bandit settings.
These signiﬁcant results support the easy-generalizability of proposed LinReBoot .
In summary, our contributions are as follows:
• Propose LinReBoot algorithms that implement Residual Bootstrap Exploration in linear bandit problems without boundness assumption of rewards.
• Theoretically show that LinReBoot secures ˜ O
( d Ô n ) regret, matching the regret bound of the same order as those theoretical results of Linear Thompson Sampling algorithms.
• Empirically show that LinReBoot rivals or exceeds baseline algorithms and supports that LinReBoot is easy-generalizable among linear bandit problems.
Related Works.
Bootstrap-based contextual bandit algorithms design has been actively studied in the last half-decade and drawn a surge of interest from both theoretical studies and industrial practice [Elmachtoub et al., 2017, Eckles and Kaptein, 2014, Osband et al., 2016, Kveton et al., 2019b, Hao et al., 2019].
Bootstrapbased bandit algorithm design is a paradigm of sequential decision-making based on an exploration mechanism with no pre-deﬁned mean reward model.
Such paradigm enjoys a decisive advantage that engineers are free to deploy any reward model of interests without painful adaption to problem structure [Kveton et al., 2019b,a].
ReBoot [Wang et al., 2020] provided a theoretical logarithmic regret guarantee for multi-armed bandit (MAB) and empirical investigation to validate the easy generalizability of the ReBoot principle.
Our work aims to provide a theoretical guarantee for the bootstrap-based linear bandit algorithms and empirically investigate more general contextual linear bandit setting to validate the ReBoot principle.
One close related work is [Kveton et al., 2020a] which introduces perturbation of past samples for exploration under stochastic linear bandit problem.
The limitation of [Kveton et al., 2020a] is the boundness of rewards, indicating many broader classes of rewards such as
Gaussian rewards are not applicable with a theoretical guarantee.
In contrast, the proposed LinReBoot algorithms relax the boundness reward assumption and thus validate bootstrap-based bandit algorithms in wider bandit environments with a broader class of reward generation processes.
Early works about exploration in bandit problems [Abbasi-Yadkori et al., 2011, Langford and Zhang, 2007,
Dani et al., 2008] are practical but no guarantee of the optimality.
Some works [Wang et al., 2020, Kveton et al., 2019b,a, Thompson, 1933, Auer et al., 2002] provide well designed exploration for bandit problems and have their own principles for adopting to more general problems.
In these works, three principles including
ReBoot [Wang et al., 2020], GIRO [Kveton et al., 2019b] and PHE [Kveton et al., 2019a] are devising exploration mechanism based on up-to-now history instead of on pre-deﬁned reward model in the other two principles
TS [Thompson, 1933] and UCB [Auer et al., 2002].
Our work generalizes ReBoot into stochastic linear bandit problems.
Notations.
Let [ n ] be set { 1 , 2 , ..., n } .
1 is a vector with all ones and I is the identity matrix.
For a vector v , Î v Î 2 is 2 -norm of v and Î v Î 2 A :=
Ô v € Av for a semideﬁnite matrix A .
Let È · , · Í be the inner product operation.
Denote F t as the history of randomness up to round t .
E t [ · ] := E [ ·|F t ≠ 1 ] is deﬁned as the conditional expectation given F t ≠ 1 and P t ( · ) :
= P ( ·|F t ≠ 1 ) is deﬁned as the conditional probability given
F t ≠ 1 .
I {·} is indicator function.
For a set or event E , we denote its complement as ¯ E .
N ( µ, ‡ 2 ) is Gaussian distribution with mean µ and variance ‡ 2 .
We use ˜ O for big O notation up to logarithmic factor.
STOCHASTIC LINEAR BANDIT
Contextualize Action Set.
In stochastic linear bandit problem, we identify the actions with d ≠ dimensional features from A µ R d and assume |A| , the size of the action set, is ﬁnite.
Let K :=
|A| be the number of actions (arms), x k œ R d be the context vector of the k -th arm, that is, A
= { x 1 , ..., x K } .
Reward generating mechanism.
The reward function is parameterized by ◊ œ R d such that, at time t the agent chooses an action I t œ [ K ] with feature
X t
= x
I t œ A , the reward is generated by
[Equation]
Speciﬁcally, the reward obtained by the agent at round t when pulling arm I t =
k is generated from a distribution with mean µ k := x € k ◊ , conditioning on context x k .
The property of noise ‘ t is described in Assumption 2.
Furthermore, denote the recieved reward by r I t and the reward random variable by Y t at round t .
Regret.
Without loss of generality, assume that arm 1 is the unique optimal arm, that is µ 1 > µ k ’ k ” = 1 .
The optimal gap of the k -th arm is  k :=
µ 1 ≠ µ k Ø 0 .
The expected n -round regret is denoted as
[Equation]
The goal of the agent is to maximize the expected cumulative reward in n rounds, which is equivalent to minimizing the expected regret R n .
Assumption 1.
(Boundness assumptions)
True parameter ◊ is bounded: Î [Equation]
Besides, we denote L as the upper bound for context vectors: Î x k Î 2 Æ L for all k œ [ K ] .
Assumption 1 is referred to the boundness assumptions in the stochastic linear bandit literature and is to ensure the regret is bounded if the agent pulls any sub-optimal actions (see Section 5 in [Abbasi-Yadkori et al., 2011]).
Assumption 2.
(Noise Clipping assumption)
Noise process { ‘ t } Œ t =1 described in (1) satisﬁes that for some
L 1 , L 2 > 0 ,
[Equation]
(3) where F t ≠ 1 = { ‘ 1 , I 1 , · · · , ‘ t ≠ 1 , I t ≠ 1 } .
Assumption 2 implies that stochastic process { ‘ t } Œ t = is conditionally sub-gaussian with constant L 2 .
Lcontributes to the lower bound of moment generating function suggested by [Zhang and Zhou, 2020].
Note that the Assumption 2 allows heteroscedasticity among di  erent arms by choosing L 2 as the largest variance among arms.
Such heteroscedasticity consideration arises and has been identiﬁed as a challenge in applications of Bayesian optimization [Kirschner, 2021,
Cowen-Rivers et al., 2020].
RESIDUAL BOOTSTRAP EXPLORATION 3. REBOOT PRINCIPLE
This section presents essential proof of concepts to implement ReBoot principle [Wang et al., 2020].
In general, each round of interaction, the decision policy admits four subroutines to implement ReBoot principle: 1) Learning, 2) Fitting, 3) Bootstrapping, and 4)
Exploring.
Following elaborates on each subroutine:
1) Model Learning.
The ﬁrst subroutine outputs a learned model based on current collected data.
Our implementation learns the parameter ◊ in Eq.(1) by some user-speciﬁed model.
2) Data Fitting.
The second subroutine ﬁts the current data set with the learned model in the previous subroutine and then outputs the residual set.
Intuitively, the residuals measure the goodness of ﬁt of the learned model and should drop a hint on the right amount of exploration.
In other words, the residuals should suggest a right magnitude of exploration bonus in decision policy (8).
How to manage and integrate uncertainty behind residuals into the exploration mechanism of policy is the main challenge.
3) Residuals Bootstraping.
The third subroutine associates the residuals obtained the last subroutine with a bootstrapping distribution.
Instead of maintaining a belief distribution on a parameter in the Bayesian approach, ReBoot principle maintains a bootstrapping distribution on the statistical error based on residuals.
The challenge is to justify the e  cacy of residual-based optimism construction in both theory and practice.
4) Actions Exploring.
The fourth subroutines sample the exploration bonus from the bootstrapping distribution and output an index for each action.
Such bootstrap procedure is more computationally e  cient than prior e  orts since this procedure only requires drawing a sample from the bootstrapping distribution.
The challenge is to prove that such bootstrap procedure secures sub-linear regret in theory.
3. LINREBOOT ALGORITHM
We propose the Linear Residual Bootstrap Exploration algorithm ( LinReBoot , Algorithm 1) for stochastic linear bandit problems.
This section elaborates the four subroutines in Section 3.1 for the proposed LinReBoot .
1) LinReBoot uses ridge regression procedure, whose learned parameter is ˆ ◊ t (4b) and estimated mean reward for arm k is ˆ µ k,t (4c).
Such way to estimate mean reward is easy to manage the conﬁdence [AbbasiYadkori et al., 2011].
Thus, we focus on conﬁdence management for the bootstrap-based exploration.
Ridge Regression Procedure.
LinReBoot ﬁts linear model at round t as follow,
[Equation]
where X t ≠ 1 =
( X 1 , ..., X t ≠ 1 ) € œ R ( t ≠ 1) ◊ d .
The · - th row of X t ≠ 1 is the context X € · for · œ [ t ≠ 1] ,
Y t ≠ 1 =
( Y 1 , ..., Y t ≠ 1 ) € is reward vector whose elements are rewards up to round t ≠ 1 .
⁄ denotes the regularization level.
V t denotes the sample covariance matrix up to round t and ˆ ◊ t is the ridge estimation of target parameter ◊ in (1).
ˆ µ k,t denotes the estimated mean of arm k based on history.
Note that the ﬁrst K rounds in proposed LinReBoot is fully exploring each arm once.
In other words, I t =
t when t œ [ K ] , indicating
X K :=
( x 1 , ..., x K ) € œ R K ◊ d .
We call this X K the context matrix with rank r Æ min( K, d ) and singular values ‡ 1 , ..., ‡ r .
Also deﬁne ‡ 2min Æ ‡ 2 i Æ ‡ 2max , ’ i œ [ r ] .
With these deﬁnitions, we make a mild assumption about the shrinkage e  ect of ridge regression:
Assumption 3.
(Validity of Ridge Regression)
The singular value decomposition of context matrix X K is denoted as X K :=
G  U where G œ R K ◊ K ,  œ R K ◊ d and U œ R d ◊ d .
Deﬁne  :=
(  €  + ⁄ I ) ≠ 1  € œ R K ◊ K and Z :=
G  U œ R K ◊ d .
Let z 1 œ R d be the ﬁrst row of Z .
Given any ⁄ > 0 , there exists a corresponding positive scalar S 1 such that | x € 1 ◊ ≠ z € 1 ◊ | Ø S 1 for the ◊ in (1) .
Remark 1.
Assumption 3 provides a lower bound of the absolute di  erence between true mean x € 1 ◊ and normalized mean z € 1 ◊ of the optimal arm.
Note that if ⁄ æ 0 , then z 1 æ x 1 and S 1 æ 0 .
Thus this scalar
S 1 measures the small perturbation on the mean of the optimal arm when the ridge regression procedure is applied.
This Z can be interpreted as a ridge shrinkage context matrix [Goldstein and Smith, 1974].
One important phenomenon of online ridge regression is that even if the ridge estimator is biased, the shrinkage e  ect from ridge estimation provides exploration for the agent leading to making a correct decision.
The positive scalar S 1 describes the shrinkage e  ect on the context.
That is, the existence of S 1 indicates the ridge procedure is valid and its shrinkage e  ect exists.
The ﬁtting part of LinReBoot outputs the residuals under the linear model framework,
[Equation]
where s k,t ≠ 1 := q t ≠ 1 · =1 I { I · = k } is the number of times pulling arm k by round t ≠ 1 , r k,i is the i -th reward of arm k by round t ≠ 1 .
The goodness of ﬁt of the learned ridge regression model can be summarised by Residual Sum of Squares(RSS)
[Archdeacon, 1994] which is deﬁned as
[Equation]
Such measure plays an important role in the residual bootstrap exploration mechanism.
The third part is Residuals Bootstrapping.
This subroutine is independent of the model which suggests the power of generalizability of ReBoot principle.
ReBoot principle requires the computation of the exploration bonus [Mammen, 1993], which is s ≠ 1 k,t ≠q s k,t ≠ 1 i =
{+Ê k,t,i e k,t,i , where { Ê k,t,i } s k,t ≠ 1 i = is residual bootstrap weights for arm k at round t . Algorithm 1 LinReBoot Require: ⁄ , s 1 , 
+}
= ... =
s K, 0 =for t = 1 , ..., n do if t < K + 1 then
I t Ω t else [Equation] for k = 1 , ..., K do e k,t,i Ω r k,i ≠ x € k ˆ ◊ t , ’ i œ { s k,t ≠ 1 } Generate { Ê k,t,i } s k,t ≠ 1 i = ˜ µ k Ω x € k ˆ ◊ t + s ≠ 1 k, t ≠q s k,t ≠ 1 i = Ê k,t,i e k,t,i end for
I t Ω œ k end for
Choice of Bootstrapping Weights.
The bootstrap weights considered in this work are i.i.d with zero mean and variance ‡ 2 Ê .
They are independent of the noise process { ‘ t } Œ t =1 .
In the literature of bootstrap procedure [Mammen, 1993] , the choices of bootstrap weights distribution include Gaussian weights, Rademacher weights and skew correcting weights.
In LinReBoot , we adopt the Gaussian bootstrap weights to enable an e  cient implement described at section 3.3.
The last subroutine is the action exploring based on residual bootstrap.
More speciﬁcally, for arm k at round t , LinReBoot adds exploration bonus from residual bootstrapping on the estimated mean ˆ µ k,t as follow,
[Equation]
then agent pulls arm with the highest bootstrapped mean,
[-Require: ⁄ , s 1 , 
-]
[Equation]
Note that the variance of bootstrapped mean ˜ µ k,t is ‡ 2 Ê s ≠ 2 k,t ≠ 1 RSS k,t , indicating an adaptive amount of extra exploration is controlled by s k,t ≠ 1 and RSS k,t .
Short Summary.
Our proposed LinReBoot has following steps at round t > K ,
1) Ridge estimation: compute V t , ˆ ◊ t . 2)
Finding residuals for each arm: for arm k , compute ˆ µ k,t and { e k,t,i [Table] [Equation] [Table]
[Equation] [Table] 3) Compute Bootstrapped mean for each arm: for arm k , generate { Ê k,t,i } s k,t ≠ 1 i = and compute ˜ µ k,t (7).
4) Pull arm with the highest ˜ µ k,t then observe reward.
Algorithm 1 describes LinReBoot .
The strength of
LinReBoot is its easy generalizability across di  erent bandit problems including linear bandits and even more complicated structured problems (Appendix D.1).
Remark 2.
( LinTS perturbs system parameter estimate, LinReBoot perturbs expected reward estimates)
Compare with the LinTS in [Agrawal and Goyal, 2013b], in which LinTS samples a perturbed parameter ˜
◊ LinTS t =
ˆ ◊ t + — [Equation]
V ≠ 1 / 2 t ÷ t with scaling — t and appropriate independent noise ÷ t (deﬁned in [Agrawal and Goyal, 2013b]).
Our proposed LinReBoot samples a perturbed expected reward
[Equation]
That is, LinReBoot is perturbing the expected reward estimate via prediction error uncertainty, which is supervised by real reward.
In contrast, LinTS is perturbing the system parameter, when can be wrong if the system modeling is wrong.
3. EFFICIENT IMPLEMENTATION
By the attractive computational properties of Gaussian distribution, the computational cost of LinReBoot can be reduced signiﬁcantly when Gaussian Bootstrap weights are generated.
Formally: assume Ê k,t,i ≥
N (0 , ‡ 2 Ê ) , ’ k, t, i , recalling (7), for k œ [ K ] and any t Ø 1 , bootstrapped mean ˜ µ k,t follows a Gaussian distribution,
[Equation]
Such Gaussian-distributed property of ˜ µ k,t indicates that if we can update ˆ µ k,t , s k,t ≠ 1 and RSS k,t incrementally for arm k , this bootstrapped mean ˜ µ k,t can be generated by Gaussian generator without inner loop for generating weights.
The ﬁrst two terms, ˆ µ k,t and s k,t ≠ 1 , are naturally updated in incremental manner.
For RSS k,t , following decomposition ensures an incremental update,
[Equation]
Then an e  cient generation for ˜ µ k,t |F t ≠ 1 is ensured by the incremental updates for ˆ µ k,t , s k,t ≠ 1 ,
[Equation]
r k,i .
Furthermore, since the residual bootstrap weights are generated independently, ˜ µ k,t among arms are also independent given historical randomness and can be sampled from one multivariate Gaussian generation simultaneously.
Formally, ˜ µ ( t ) = (˜ µ 1 ,t , . . . ,
˜ µ K,t ) € is conditional distributed as
[Equation]
where ˆ µ ( t )
= (ˆ µ 1 ,t , . .
, ˆ µ K,t ) € and  ( t ) Ê is a diagonal matrix with diagonal elements ‡ 2 Ê s ≠ 2 k,t ≠ 1 RSS k,t .
Detailed steps and more illustration about e  cient implementation is provided in Appendix D.7.1.
Moreover, an empirical study about computational e  ciency is conducted in Appendix D.7.2 and Table.3 provides the computational cost of our proposed LinReBoot as well as other baseline algorithms.
OPTIMISM DESIGN
Optimistic Estimated Discrepancy.
This section identiﬁes and demystiﬁes the technical challenge of implementing ReBoot principle in the stochastic linear bandit problem.
The key is to conduct a detailed investigation to produce probabilistic control on the behavior of the ’ O ptimistic E stimate D iscrepancy ( OED )’ of the LinReBoot policy (8).
In principle, the OED is given by
[Equation]
where the Action Context Norm is given by Î x k Î V ≠ 1 t and Optimism is given by c t,k for the k th action at time t , deﬁned in (14).
Design of c t,k will be elaborated in Section 4.1.
Su  cient Explored Arms.
We deﬁne the concept of
Su  cient Explore Arms to facilitate the formal regret analysis of LinReBoot .
Intuitively, an arm is su  cient explored if its index produced by the policy (8) is less than the mean reward of the optimal arm.
Technically, we say an arm k is su  ciently explored at time t if the adopted OED ( c t,k Î x k Î V ≠ 1 t ) is bounded by its optimal gap (  k ).
The above notion of su  cient explored arm deﬁnes the concept of ”set of su  cient explored arms” S t , formally
[Equation]
where and c t,k is the collaborated optimism and c t,k Î x k Î
[Equation]
t is an optimistic estimate of discrepancy of policy index (8).
The key consequence of set (12) is that, any member in S t enjoys the property
[Equation]
that is, the LinReBoot policy always avoids an index (8) from su  ciently explored subset such that the bootstrapped mean of this index is less than the optimal mean reward unless all arm are su  ciently explored.
(see equation (82) in the proof of Lemma A.1 at section
B.1 for technical details).
4. COLLABORATE OPTIMISM
Here we elaborate on the collaborated optimism adopted in the deﬁnition of su  cient explored arms (12).
Concretely, the collaborated optimism has a form
[Equation]
where c 1 ( t, k ) is called sample optimism and c 2 ( t, k ) is called bootstrap optimism for arm k at time t .
Sample Optimism.
The sample optimism c 1 ( t, k ) serves as a control on the event that ”the realized sample estimate discrepancy (ED) is bounded by sample
OED”:
[Equation]
where c 1 ( t, k ) is a constant which can be tuned by
our
LinReBoot algorithm, making the bad event ¯ E t,k and ¯
E become unlikely.
In fact, this E t,k is the event that the least squared estimation is "close" to the true mean reward for arm k at round t .
In section 5, the probability of the bad event ¯ E t is controlled by a parameter tuned by users based on lemma 5.1.
Bootstrap Optimism.
The bootstrap optimism c 2 ( t, k ) serves as a control on the event that ”the realized bootstrap ED is bounded by bootstrap OED”:
[Equation]
where c 2 ( t, k ) is also a constant controlling the conditional probability of the bad event ¯ E Õ t .
This c 2 ( t, k ) can be tuned by our LinReBoot algorithm as well.
Similar to E t,k , this E Õ t,k is the event that the residual bootstrap based estimation is "close" to the least squared estimate ˆ µ k,t for arm k at round t .
In section 5, the probability of bad event ¯ E Õ t is controlled by a parameter tuned by users based on lemma 5.2.
4. OPTIMISM DESIGN
Choice of sample optimism ( – ).
The goal of this part is to illustrate how to pick the sample OED such that the event (15) holds with probability at least 1 ≠ – for a given conﬁdence budget – œ (0 , 1) .
Formally, the goal is to ﬁnd a sample OED function c 1 ( t, k ) :
[ n ] ◊ [ K ] ‘æ R such that the event (15a) holds with probability at least 1 ≠ – k .
To meet the purpose of the risk control, we specify the sample OED function with form
[Equation]
Lemma 5.1 gives the formal result on why such choice has conﬁdence budget at most – k .
For regret analysis, deﬁne –
[Equation]
k
[Equation]
Choice of bootstrap optimism ( — ).
The goal of this part is to pick bootstrapped OED such that the event (16) holds with probability at least 1 ≠ — for given conﬁdence budget — œ (0 , 1) .
Formally, the goal is to ﬁnd a sample OED function c 2 ( t, k ) :
[ n ] ◊ [ K ] ‘æ R such that the event (16a) holds with probability at least 1 ≠ — k .
To meet the purpose of the risk control, we specify the bootstrapped OED function with form
[Equation]
Lemma 5.2 gives the formal result on why such choice has a conﬁdence budget at most — k .
For regret analysis, let — min be the smallest — k , ’ k œ [ K ] and — = ( — 1 , ..., — K ) € .
4. OPTIMISM FOR OPTIMAL ARM
Sample-Bootstrap OED ratio of the optimal arm (b).
Indicated by the regret analysis in [Kveton et al., 2020a], instead of controlling the exploration independently, the relation between two sources of explorations needs to be considered because this relation is critical for ﬁnding the optimal action.
To meet such observation, we deﬁne a good event,
[Equation]
Given the good event E ÕÕ t , the policy index ˜ µ 1 ,t of the optimal arm enjoys further positive bias, hence the agent will have better chance to make optimal action.
In particular, we highlight a constant b used to measure the ratio of the sample optimism (17) to the bootstrap optimism (18); formally, we require b satisﬁes
[Equation]
Intuitively, the constant b measures the relation between sample OED and bootstrap OED of the optimal arm.
This b plays an important role of the probability lower bound of event (19) (See Lemma 5.3).
Note that, if (20) holds, we have the lower bound (26) ; otherwise, we have the lower bound (27).
In both cases, we have a lower bound for the event (19). Notation Deﬁnition
[Equation]
Good event for optimal arm ( “ ).
Here we introduce the event that over exploration and under exploration of the optimal arm have been avoided simultaneously.
Formally, the constant “ is the probability that the bandit index (8) is not over-exploration (Event E Õ t ) and also not under-exploration (Event E ÕÕ t )
[Equation]
Technically [Equation] we can show that the probability of the event (21) is lower bounded by the term
[Equation]
with probability at least 1 ≠ “ (Lemma 5.4).
Such lower bound is translated into an upper bound in regret analysis.
FORMAL RESULTS 5. REGRET BOUND FOR LINREBOOT
Theorem 5.1.
Under Assumptions 1, 2, 3 and technical conditions (32) and (74) , with probability at least 1 ≠ ( ” + “ ) , the expected regret of Algorithm 1 is bounded as, R n Æ C 1 ( – 1 , — , “ , b )
[Equation]
1 ( n, d )
[Equation]
[Equation]
[Equation]
n
[Equation]
d
[Equation] [Equation] [Equation] 23) where ’ 1 , ’ 2 , ’ 3 and ’ 4 are deﬁned in Table.1 and C 1 ,
C 2 , M 1 , M 2 are described in Table.2.
Proof.
See Appendix A.1.
Corollary 5.2.
Let – = — = 1 Ô n 1 , the order of high probability upper bound in Theorem 5.1 is ˜ O
( d Ô n ) .
Proof.
See Appendix A.2.
Corollary 5.2 shows that our regret bound scales as the regret bound of Linear Thompson
sampling [Agrawal and Goyal, 2013b] and Linear PHE [Kveton et al., 2020a].
5. VALIDATE SAMPLE OPTIMISM
Lemma 5.1.
Under Assumptions 1, 2, 3 and choose c 1 ( t, k ) as (17) , P ( ¯ E t,k ) , the probability of bad event corresponded to least squared estimation described in (15), is controlled.
Formally, ’ k œ [ K ] , ’ – k > 0 , ’ t Ø 1 ,
[Equation]
Consequently, we have P ( ¯ E t ) Æ – := q Kk =1 – k .
Proof.
See Appendix A.3.
Lemma 5.1 supports that the choice of c 1 ( t, k ) at (17) for the sample optimism event (15) is valid with conﬁdence budget – .
5. VALIDATE BOOTSTRAP OPTIMISM
Lemma 5.2.
Suppose bootstrap weights are Gaussian.
Pick c 2 ( t, k ) as (18) .
The conditional probability of bad event corresponding to residual bootstrap exploration described in (16) , P t ( ¯ E Õ t,k ) , is controlled.
Formally, ’ k œ [ K ] , ’ — k > 0 , ’ t Ø
P
[Equation] [Equation]
Consequently, we have P t ( ¯ E Õ t )
[Equation]
[Equation]
Proof.
See Appendix A.4.
Lemma 5.2 supports that the choice of c 2 ( t, k ) at (18) for the sample optimism event (16) is valid with conﬁdence budget — .
5. SAMPLE-BOOTSTRAP RATIO
Lemma 5.3.
Under Assumptions 1, 2, 3.
Suppose bootstrap weights are Gaussian.
The conditional probability of anti-concentration for optimal arm described in (19), P t ( ¯ E ÕÕ t ) , has lower bound.
Formally, if b satisﬁes (20) ,
[Equation]
Otherwise,
[Equation]
where  is the CDF of standard normal distribution.
Proof.
See Appendix A.5.
Lemma 5.3 provides the lower bound result for good event E ÕÕ t .
The result indicates that, if the bootstrap optimism is not ’too large’, then the LinReBoot procedure can enjoy additional regret reduction.
5. VALIDATE GOOD EVENT
Lemma 5.4.
Under Assumptions 1, 2, 3 and suppose
Bootstrap weights are Gaussian.
Assume b satisﬁes a technical condition (74) .
Then, with probability at least 1 ≠ “ , P t ( E ÕÕ t )
≠ P t ( ¯ E [Equation] t ) has lower bound,
[Equation]
where M 1 and M 2 are deﬁned in Table.2.
Proof.
See Appendix A.6.
Lemma 5.4 provided the a high probability lower bound for the di  erence between probability of the event for anti-concentration E ÕÕ t and probability of bad event discussed in bootstrap optimism in Section 4.1.
This lower bound is also for probability of ‘not under and not over exploration’ event (21).
Lemma 5.4 links the sample optimism and bootstrap optimism and holds a right amount of exploration of the optimal arm.
EXPERIMENTS
In this section, we conduct empirical studies under three settings: Stochastic Linear Bandit, Contextual
Linear Bandit and Linear Bandit with Covariates.
Our
LinReBoot is compared to several baselines including
LinTS-G [Agrawal and Goyal, 2013b, Lattimore and
Szepesvári, 2020], LinTS-IG [Honda and Takemura, 2014, Riquelme et al., 2018], LinPHE [Kveton et al., 2020a], LinGIRO [Kveton et al., 2019b] and LinUCB [Abbasi-Yadkori et al., 2011, Lattimore and Szepesvári, 2020] .
More details about baselines can be found in
Appendix D.6.
6. STOCHASTIC LINEAR BANDIT
We compare LinReBoot to other linear bandit algorithms under stochastic linear bandit described in Section 2.
We experiment with several dimensions d including 5 , 10 and 20 .
K is chosen as 100 .
Synthetic data generation for this setting is deferred to Appendix
D.2 in the supplementary material.
Results.
The ﬁrst row of Figure 1 reports the results for Stochastic Linear Bandit setting.
Our LinReBoot rivals LinTS-G and
LinTS-IG while substantially exceeds LinGIRO , LinPHE and LinUCB .
When d increases, the performance of
LinReBoot rivals and exceeds the best of other methods.
6. CONTEXTUAL LINEAR BANDIT
In the second experiment, we compare LinReBoot to other linear bandit algorithms under Contextual Linear
Bandit where the contexts are generated from some distributions by arms.
Note that this setting matches previous work [Chu et al., 2011].
Linear bandit algorithms can also be applied under this kind of environment.
In our experiment, the LinReBoot is implemented as
Algorithm 2 in Appendix D.1.
Like the setting in Section 6.1, the dimension of d is chosen as 5 or 10 orand the synthetic data generation for this setting is described in Appendix D.2.
Results.
The second row of Figure 1 reports the results for Contextual Linear
Bandit.
Our LinReBoot rival LinTS-G and substantially exceed LinTS-IG , LinGIRO , LinPHE and LinUCB .
When d increases, the performance of LinReBoot rivals
LinTS-IG and exceeds others.
6. BANDIT WITH COVARIATES
Our last experiment is conducted under the setting of linear bandit with covariates, which is also called linear parametrized bandit by [Rusmevichientong and Tsitsiklis, 2010].
This problem is signiﬁcantly di  erent from the previous two problems in the following ways.
Each arm has its true parameter ◊ k .
That is, each arm has its estimate ˆ ◊ k from the ridge regression procedure in
Section 3.2.
Also, unlike the setting in Section 6.2, the contexts are generated from a distribution that is independent of arms.
Thus the overall task in this setting is not only the estimation of the target parameter ◊ , but also the detection of which arm a context belongs to.
This case is also referred to as the online decisionmaking under covariates [Bastani and Bayati, 2020].
For the LinReBoot in this setting, detailed algorithm is provided as Algorithm 3 in Appendix D.1.
d is chosen as 5 or 10 or 20 and K = 10 .
Synthetic data generation for this setting is described in Appendix D.2.
Results.
The third row of Figure 1 reports the results for Linear Bandit with Covariates.
Our LinReBoot exceeds all competing algorithms LinTS-G , LinTS-IG , LinGIRO ,
LinPHE and LinUCB .
Summary.
From Figure 1, the proposed LinReBoot is always the top 3 algorithms under all settings and all choice of dimension d .
More speciﬁcally, LinReBoot is clearly comparable to the state-of-the-art Linear
Thompson Sampling algorithms( LinTS-G , LinTS-IG ) or even outperforms them in many cases.
Regarding the computational cost, from Table.3, our proposed LinReBoot is consistently computational e  cient among all settings compared to LinTS-G , LinTS-IG and
LinUCB under all three settings.
CONCLUSION
We propose LinReBoot algorithm for stochastic linear bandit problems.
In theory, we prove LinReBoot that secures ˜ O ( d Ô n ) high probability expected regret.
Empirically, we show LinReBoot rivals LinTS-G , LinTS-IG and exceeds LinPHE , LinGIRO and LinUCB , which supports the easy-generalizability of ReBoot principle in [Wang et al., 2020] under various contextual bandit settings including Stochastic Linear Bandit, Contextual
Linear Bandit, and Linear Bandit with Covariates.