Causal Bandits: Online Decision-Making in Endogenous Settings
Abstract
The deployment of Multi-Armed Bandits (MAB) has become commonplace in many economic applications.
However, regret guarantees for even state-of-the-art linear bandit algorithms (such as Optimism in the Face of Uncertainty Linear bandit (OFUL)) make strong exogeneity assumptions w.r.t.
arm covariates.
This assumption is very often violated in many economic contexts and using such algorithms can lead to sub-optimal decisions.
Further, in social science analysis, it is also important to understand the asymptotic distribution of estimated parameters.
To this end, in this paper, we consider the problem of online learning in linear stochastic contextual bandit problems with endogenous covariates.
We propose an algorithm we term ϵ - BanditIV , that uses instrumental variables to correct for this bias, and prove an ˜ O ( k √ T ) 1 upper bound for the expected regret of the algorithm.
Further, we demonstrate the asymptotic consistency and normality of the ϵ - BanditIV estimator.
We carry out extensive Monte Carlo simulations to demonstrate the performance of our algorithms compared to other methods.
We show that ϵ - BanditIV significantly outperforms other existing methods in endogeneous settings.
Introduction
The proliferation of user-level data presents at the same time a unique opportunity and challenge in front of decision-makers.
Decision-makers want to use individual-level data to tailor their decisions for each user.
Further, given the dynamic nature of the platform, decision-makers want to be able to adopt these decisions with incrementally available data.
Woodroofe 1979 first proposed a simple model to solve such sequential decision-making problems with covariates.
Langford and T. Zhang 2007 later named the model "contextual bandit".
Contextual linear bandits have been adopted across a wide variety of applications from advertising (Tang et al. 2015, Aramayo, Schiappacasse, and
Goic 2022) to healthcare (Durand et al. 2018), dialogue systems (Liu et al. 2018), and personalized product recommendations (L. Li et al. 2010, Qin, S. Chen, and Zhu 2014).
In the setting of a contextual linear bandit problem, in each round, the decision-maker observes a set of actions with each action characterized by a set of features.
Decision-maker selects an action and observes a reward corresponding to that action.
The objective of the decision-maker is to achieve cumulative reward close to that of optimal policy in hindsight.
Traditional formulations of contextual bandits make the unconfoundedness assumption i.e., the arm covariates are exogenous.
However, in many economic settings, arm features can be correlated with the unobserved noise.
For instance, consider the problem of generating product recommendations for consumers.
Platform operators would usually run online experiments to uncover the relationship between product features and consumer demand.
Generally in such settings, product observable features like price are controlled by product owners (different from the platform operator) which could be set in anticipation of consumer demand and hence be correlated with the demand shocks unobserved by the platform operator.
In such settings, traditional bandit algorithms might lead to sub-optimal decisions.
A common approach to correct the endogeneity bias in the offline setting is to use instrumental variables.
Instruments are correlated with the endogenous variable but are otherwise not associated with the outcome variable.
The method of instrumental variables uses the variation in the exogenous component of the endogenous variable induced by the variation in the instrumental variable to make inference of causal effects.
Thus, to address the issues induced by endogenous features in online settings, we propose an online estimator we term BanditIV that uses instrumental variables to learn the relationship between rewards and features.
Next, literature in contextual linear bandit has also primarily focused on minimizing the expected regret of the algorithm (Auer 2002, Dani, Hayes, and Kakade 2008, Chu et al. 2011, and AbbasiYadkori, Pál, and Szepesvári 2011).
However, in a variety of practical situations, one may also be interested in the inference of model parameters, and algorithms that minimize regret may not guarantee the consistency of the model parameter estimator.
Further, knowing the asymptotic distributions, one can easily test the significance of features, and offers a traditional way to select variables.
For instance, in the product recommendation example, an online platform might also be interested in understanding the effect of various marketing-mix variables (like price, and promotion) on consumer demand.
To this end, we propose a linear bandit algorithm that pursues both regret minimization and consistent estimation of model parameters in endogenous settings.
We use the martingale central limit theorem to show that our estimator of model parameters is asymptotically normal.
Finally, we compare our method with existing online algorithms including Optimism in the Face of
Uncertainty Linear bandit (OFUL) and Thompson Sampling (TS) through numerical experiments.
We find significant advantages of our algorithm in both expected regret and parameter inference.
To summarize, our paper makes the following contributions –
• We study the stochastic linear bandit problem with endogenous features.
We propose a new algorithm, we term BanditIV , which incorporates instrumental variables to correct for the bias induced by endogenous features and show that the total expected regret of the algorithm is upper bounded by ˜ O
( k √ T ) .
• Next, as in many economic contexts, researchers might not only be interested in minimizing the regret over the outcomes but also in conducting inference over estimated parameters.
To this end, we propose the ϵ - BanditIV algorithm with the same upper bound and establish the asymptotic consistency and normality of the estimator.
Finally, we conduct numerical experiments of BanditIV and ϵ - BanditIV algorithms on synthetic data.
We show BanditIV and ϵ - BanditIV always outperform other existing linear bandit algorithms including OFUL and TS in terms of both total expected regret and inference.
To see a complete version of literature review, please refer Appendix A.
Notation.
Throughout this paper, we use || · || p to denote the p -norm of a vector or a matrix.
We use || · || F to denote the Frobenius norm for a matrix.
For a vector x and positive definite matrix A , we
[Equation]
denote √ x ′
Ax as || x
|| A .
We use ⟨· , ·⟩ as the inner product.
We denote the sequence a 0 , a 1 , · · · , a ∞ as { a t } ∞ t =0 .
For a matrix X , the i th column is denoted as X ( i ) and the j th element of the i th column is denoted as X ( i )( j ) .
Problem setting
Let T be the number of rounds and K the number of arms in each round.
In each round t , the learner observes K feature vectors x t,a ∈ R d , a ∈ { 1 , · · · , K } , with || x t,a || 2 ≤ L x for samples (one feature vector for each sample), where the subscript a represents the a th arm or sample from the K -sample set.
Let y t,a ∈ R , s.t.
| y t,a | ≤ L y , be the reward of arm a on round t .
After observing the feature vectors, the learner selects an arm a t and receives reward y t,a .
Under linear realizability assumption, we have E [ y t,a | x t,a ] = x ′ t,a β 0 for all t and a , where β 0 ∈ R
d is an unknown true coefficient vector.
We assume y t,a are independent random variables with expectation x ′ t,a β 0 , as the following.
[Equation]
where e t,a ∈ R is a 1-subgaussian error term, s.t.
E [ e t,a ] = 0 .
Standard Contextual Linear Bandit settings have E [ e t,a x t,a ] = 0 , which assumes that the error term is independent with the feature vector.
However, this assumption can be violated in cases such as when we have omitted variable in the error term, measurement error in the regressor, simultaneous equation estimation, and etc.
Mathematically, when we have the following holds,
[Equation]
the endogeneity problem occurs.
Ordinary Least Square (OLS) estimator is inconsistent in endogenous settings.
Econometrics literature uses Instrumental Variable (IV) method to address the endogeneity problem.
A variable z t,a is a valid instrumental variable if it satisfies following three conditions (M Wooldridge 2014): (i) it is uncorrelated with the error term e t,a , i.e. E [ z t,a e t,a ] = 0 , (ii) it is correlated with the endogenous covariates x t,a , (iii) it has no direct effect on the reward y t,a .
Consider a valid instrumental variable z t,a ∈ R k , with || z t,a || 2 ≤ L z , and the minimum eigenvalue of E [ z t,a z ′ t,a ] is positive.
We assume E [ z t,a x ′ t,a ] ∈ R
k × d has full column rank d .[-
-]
[Equation]
where Γ 0 ∈ R k × d is an unknown true coefficient vector, u t,a ∈ R d is a 1-subgaussian error term with mean zero and the independency condition E [ z t,a u ′ t,a ] = 0 is satisfied by construction.
By plugging the equation of x t,a (Equation 2) into Equation 1, we have y t,a = (Γ 0 β 0 ) ′
z t,a + v t,a where v t,a = β ′
u t,a + e t,a .
We denote Γ 0 β 0 as δ 0 .
Recall that in offline settings, we can use instrumental variables to solve endogeneity problem through methods like Two-Stage Least-Squares (TSLS).
Consider vector Y ∈ R T with y t,a as its elements, matrix X ∈ R T × d with x ′ t,a as its rows, matrix Z ∈ R T × k with z ′ t,a as its rows, where t ∈ { 1 , · · · , T } .
We briefly illustrate TSLS estimation procedure in offline settings as the following: (i)
First, regress the set of feature vectors X on the set of instrumental variable vectors Z using OLS method to obtain an estimated sample features ˆ X ; (ii) Then regress Y on the estimated feature vectors ˆ X to obtain the estimator ˆ β ˆ X,Y using OLS method again.
Definition 1.
(Two-Stage Least Squares Estimator)
[Equation]
To facilitate further analysis on the TSLS estimator, we also define two related OLS estimators, ˆΓ =
( Z ′ Z ) − 1 Z ′ X
and ˆ δ =
( Z ′ Z ) − 1 Z ′ Y where ˆ δ =
ˆΓˆ β ˆ X,Y (to see this equation, refer Appendix B for the proof).
It’s known that TSLS estimator is consistent in offline settings, while it remains unknown for the consistency of the TSLS estimator in online settings.
As far as we know, we are the first who combine the contextual linear bandit algorithm with TSLS and prove the consistency of TSLS estimator in online settings.
We provide the asymptotic properties in Section 4.
We aim to design an online decision making algorithm that learns the coefficient of main interest βso that we can maximize the total expected reward after pulling arms.
We define the total expected regret of an algorithm A after T rounds as
[Equation]
where a ∗ t = arg max a ⟨ Γ ′
z t,a , β 0 ⟩ is the best arm at round t according to the true coefficient vectors β 0 and Γ 0 , and a t is the arm selected by the algorithm A at round t .
We relist key assumptions in this paper as the following,
Assumption 1 .
||
z t,a || 2 ≤ L z , || x t,a || 2 ≤ L x , || y t,a || 2 ≤ L y for all t and a .
Assumption 2 .
The minimum eigenvalue of E [ z t,a z ′ t,a ] is larger than a positive constant λ .
For the simplicity of notations, we omit the subscript a of z t,a , x t,a , and y t,a from below.
By x t , we mean one sample or arm chosen by the algorithm from the arm set X t at time t ; z t is the instrumental variable related to x t ; y t is the reward generated from the arm x t . By x ∗ t , we mean the true optimal sample or arm from the arm set X t at time t ; z ∗ t is the instrumental variable related to x ∗ t .
BanditIV algorithm
To address the endogeneity problem in online settings, we propose the following BanditIV Algorithm based on existing linear bandit algorithms, especially the OFUL algorithm proposed by AbbasiYadkori, Pál, and Szepesvári 2011.
The BanditIV algorithm 3 takes as input initial regularization parameters γ z , γ x > 0 , as well as confidence interval parameters { G t } ∞ t =1 , { B t } ∞ t =1 .
We construct two confidence sets C 1 ,s , C 2 ,s , s ∈ { 1 , · · · , T } for the first stage estimation and the second stage estimation respectively in each round.
The idea for the confidence sets are to make the estimation optimistic with the condition that "with high probability" the true coefficients are in the confidence sets (Abbasi-Yadkori, Pál, and
Szepesvári 2011).
The best arm generated by the algorithm is a sample x t which is related to an instrumental variable z t that can maximize the estimated reward jointly with a pair of optimistic estimates of two-stage coefficients in the confidence sets.
If the current time t is after the first round, we utilize past choices X t and observations of rewards Y t to estimate the coefficients Γ 0 and β 0 by the TSLS estimator in each round.
We denote the estimator ˆ β t used in this algorithm as the BanditIV estimator .
Based on the estimation and confidence sets, we choose the best arm and then observe the reward.
Taking the BanditIV algorithm as a special case, we further propose the ϵ - BanditIV algorithm (which we show in Algorithm 1).
The ϵ - BanditIV algorithm additionally takes a sequence of non-increasing exploration parameters { ϵ t } ∞ t =1 as input.
In each round, instead of choosing the best arm, we conduct a stochastic decision with probability ϵ t for choosing a random arm and probability 1 − ϵ t for choosing the estimated best arm.
Notice that when ϵ t = 0 for t ∈ { 1 , 2 · · · , ∞} , the ϵ - BanditIV algorithm degenerates to the BanditIV .
For the simplicity of notations, we omit the subscript a of z t,a , x t,a , and y t,a from below.
By x t , we mean one sample or action chosen from the action set X t at time t ; z t is the IV related with x t ; y t is the reward generated from the action x t .
In this section, we give upper bounds on the regret of the BanditIV algorithm and ϵ - BanditIV algorithm. The proofs can be found in Appendix A. We show an ˜ O ( k √ T ) upper bound for the total expected regret of both algorithms with parameters of confidence set by Theorem 1.
Input: γ z , γ x > 0 , { G t } ∞ t =1 , { B t } ∞ t =1 , { ϵ t } ∞ t =
Set U
= γ z I ∈ R k × k , V 0 =
∈ R k × d , W
=
γ x I ∈ R d × d , Q 0 =
∈ R d ×
Set C 1 ,s =
{ Γ : || Γ ( i ) − ˆΓ ( i ) s
||
U s ≤ G s } , C 2 ,s = { β : || β −
ˆ β
s || W s ≤ B s } , ∀ s ∈ { 1 , 2 , · · · T } , ∀ i ∈ { 1 , 2 , · · · d }
Nature reveals Z 0 .
We randomly choose z 0 ∈ Z 0 and set Z
=
z
Nature reveals X 0 .
From the set X 0 , we play x 0 which is related with z
and then observe the reward y 0 .
We set X 0 = x 0 and Y
= y 0 for t := 1 , 2 , · · · , T , do
U t =
U t − 1 + z t − 1 z ′ t − 1 , V t =
V t − 1 + z t − 1 x ′ t − 1 ˆΓ t =
U − 1 t V t , ˆ X t − 1 = Z t − 1 ˆΓ t
W t =
{+W 
+}
+ ˆ X ′ t − 1 ˆ X t − 1 , Q t =
{+Q 
+}
+ ˆ X ′ t − 1 Y t − 1 ˆ β t = W − 1 t Q t
Nature reveals Z t .
With probability ϵ t , we uniformly choose a random z t ∈ Z t .
With probability 1 − ϵ t , we choose z t = arg max z ∈Z t max Γ ∈ C 1 ,t max β ∈ C 2 ,t ⟨ Γ ′ z, β ⟩ and update Z t =
[ Z ′ t − 1 z t ] ′
Nature reveals the set of arms X t .
We play x t which is related with z t and observe the reward y t .
Update
X t =
[ X ′ t − 1 x t ] ′ , Y t =
[ Y ′ t − 1 y t ] ′ end for
Theorem 1.
The expected cumulative regret of the ϵ -BanditIV at time T, with probability at least 1 − δ , is upper-bounded by
[Equation]
where B T = √ γ x || β 0 || 2 +
[Equation]
By setting ϵ t = 0 , we can derive the upper bound for the BanditIV directly.
Remark 1 .
Notice that in order to guarantee an ˜ O ( K √ T ) upper bound for the total expected regret of the ϵ -BanditIV algorithm, we need a small enough ϵ t , such as an ϵ t ≤ √ log( t ) √ t .
The BanditIV algorithm which is a special case with ϵ t = 0 in ϵ -BanditIV algorithm, naturally satisfies this condition.
Inference: tail inequality and normality
In this section, we show asymptotic properties of the BanditIV estimator following H. Chen, Lu, and
Song 2021 and Bastani and Bayati 2020.
We first present the consistency and then the normality of the BanditIV estimator.
The proofs are presented in Appendix A. Additionally, we calculate confidence intervals and show the results in A due to limited space.
Proposition 1.
(Tail bound for the BanditIV estimator)
In the online decision-making model with ϵ -greedy policy, if the Assumptions 1 and 2 are satisfied, and ϵ t is non-increasing, then for any η 1 , η 2 > 0 , η 2 ̸ = λ min (Γ 0 ) ( kd ) 12 , P [Equation] ||
[Equation] ≤ [Equation] β
[Equation]
≥ p 1 p dwhere C β = η
[Equation] + η [Equation]
||
[Equation]
||
[Equation] [Equation]
Remark 2 .
If tϵ 2 t → ∞ as t → ∞ , then the probability of || ˆ β t − β 0 || ≤ C β goes to 1 for any η 1 , η 2 > 0 , η 2 ̸ = λ min (Γ 0 ) ( kd ) 12 .
We need carefully choose the value for ϵ t in order to guarantee both the regret bound and the tail bound.
For example, an ϵ t = √ log( t ) √ t or ϵ t = √ loglog( t ) √ t satisfies both the conditions for the regret bound and the tail bound.
Following from the Propositions 1, we obtain the consistency of the BanditIV estimator easily.
Corollary 2.
(Consistency of the online BanditIV estimator).
If Assumptions 1 and 2 are satisfied, ϵ t is non-increasing and tϵ 2 t → ∞ as t → ∞ , then the online BanditIV estimator ˆ β t is a consistent estimator for β 0 .
Theorem 3.
(Asymptotic normality of the online BanditIV estimator)
If Assumptions 1 and 2 are satisfied, ϵ t is non-increasing and tϵ 2 t → ∞ as t → ∞ .
Then
[Equation]
where S = E [ v 2 ](Γ ′ 
 (cid:82) zz ′
d P z Γ 0 ) − 1 Γ ′
(cid:82) zz ′
d P z Γ {+
+} (Γ ′ 
 (cid:82) zz ′ d P z Γ 0 ) −
A consistent estimator for S is given by
[Equation] where ˆ v s = y s − (ˆΓ s ˆ β s ) ′ z s .
Theorem 3 provides a theoretical guarantee that the BanditIV estimator asymptotically follows a normal distribution with mean zero and variance S . We can see that this variance depends on the expectation of the error term v t , which includes the errors in both first stage and the second stage.
The variance also depends on the distribution of the instrumental variable, P z .
Notice that, although we need the assumption about ϵ t to guarantee the consistency of the estimator, the asymptotic variance of the BanditIV estimator does not depend on ϵ t .
Numerical Experiments
In this section, we construct synthetic data to further validate our algorithm.
Referring the simulation set up in Bakhitov and Singh 2021, we consider the model as follows, for t = 1 , · · · , T ,
[Equation]
Suppose that all elements of the instrument Z t ∈ R k × 1 are uniformly distributed on the support [ − 3 , 3] .
The error term e t ∈ R n t × d is the confounder, where n t is the number of arms at time t and all elements of e t follows N (0 , 1) .
The parameter ρ ∈ R d × 1 measures the degree of endogeneity.
A lower ρ implies a less serious endogeneity issue.
As an extreme example, when ρ = 0 , endogeneity disappears, which we can see from Equation (3). The additional noise terms u t ∈ R n t × d , and ε t ∈ R n t × 1 are i.i.d.
normally distributed. All elements of u t and ε t follow N (0 , 0 . 01) and N (0 , 1) respectively. WLOG, we set n t = 50 , for t = 1 , · · · , T in this simulation.
[Equation]
We have two objectives in the simulation: (i) to achieve the maximum reward through selecting optimal arms (ii) to obtain an accurate estimation of the causal relation parameter β 0 . We compare performance of our algorithm regarding the two objectives, with other existing algorithms including TS and OFUL under endogeneity.
We run T = 2000 time steps for each algorithm and observe the regret and estimation bias along the time.
We use the true cumulative regret which excludes random error terms to measure the regret and || β 0 − ˆ β t || 2 to measure the estimation bias. Figures 1 and 2 show the results where we consider various endogeneity degrees and dimensions of the instrumental variable.
In Figure 1, we set the dimension of the instrumental variable as k = 1 which is equal to the dimension of endogenous variable d = 1 . When k = d , we have the same number of instrumental variables as that of endogenous variables and β 0 will be just identified.
In Figure 2, we set the dimension of the instrumental variable as k = 2 which is larger then the dimension of endogenous variable d = 1 . When k > d , we have more instrumental variables than endogenous variables, which can cause overidentification.
Across these two cases regarding the dimensions, we find quite robust results that our proposed algorithms outperform TS and OFUL both on regret and inference.
This outperformance is more significant under higher endogeneity.
In both Figures 1 and 2, the first, second, third row present results when ρ = 2 , 1 , 0 .
5 respectively. We can see that BanditIV and ϵ -BanditIV can achieve lower bias in inference than TS and OFUL and this difference become larger when the endogeneity degree increases. Also, BanditIV reaches lower expected regret than ϵ -BanditIV, but ϵ -BanditIV can obtain a less biased estimation.
Conclusion
In this paper, we study the endogeneity problem in online decision making settings where we formulate the decision making process as a Contextual Linear Bandit model.
The existence of endogeneity can lead to estimation bias and sub-optimal decision.
To correct the bias and optimize the decision, we propose the BanditIV algorithm by utilizing both existing linear bandit algorithms in computer science literature and TSLS method in econometric literature.
We present various theoretical properties of the algorithm.
We first show an upper bound for the total expected regret, and then show the consistency, asymptotic normality of the estimator in the algorithm.
On the applied side, we conduct simulations on synthetic data.
We find that the BanditIV algorithm outperforms several benchmark linear bandit algorithms, especially when endogeneity problem occurs.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We don’t provide the code
but we do provide instructions for the data and the parameters we use in the experiments (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] We use normal computer resource.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] We don’t use existing code, data or models.
(b) Did you mention the license of the assets?
[N/A] We don’t use existing code, data or models.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] We don’t have any new asset.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We don’t use existing code, data or models.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] We don’t use existing code, data or models.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] We don’t use crowdsourcing (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] We don’t use crowdsourcing (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] We don’t use crowdsourcing
A
Appendix
Appendix A.1 Figures
[Figure]
[Figure]
Appendix A.2 Literature review
Our paper contributes to three different strands of literature (i) adaptive experimentation, recommendation engines, and advertising, (ii) interactive learning and inference with MABs, and (iii) machine learning and econometrics. Recently, there has been a surge of papers studying adaptive experimentation, recommendation, and targeted advertising. Adaptive experiments are increasingly used to investigate the effects of recommendation and targetd advertising. Araman and Caldentey 2022 studied a Bayesian sequential experimentation problem through dynamic programming and diffusion-asymptotic analysis.
K. Misra,
Schwartz, and Abernethy 2019 proposed a dynamic price experimentation policy which extended
MAB algorithms to include microeconomic choice theory.
Gur and Momeni 2022 considered auxiliary information between decision epochs into the development of an adaptive policy to improve experimentation performance.
Bertsimas, Korolko, and Weinstein 2019 proposed an online allocation algorithm in experimental clinical trials using robust mixed-integer optimization to achieve a high statistical power.
Moazeni, Defourny, and Wilczak 2020 modeled the marketing campaign performance as a multiplicative advertising exposure problem and proposed a computationally efficient learning policy through solving sequentially mixed-integer linear optimization problems.
Delshad and Khademi 2022 studied the personalized dose-finding clinical trial problem through a Bayesian framework by stochastic programming and proposed an adapted one-step look-ahead approximate policy.
Anderer, Bastani, and Silberholz 2022 introduced a Bayesian adaptive clinical trial design that combined both surrogate and true outcomes to improve trial performance.
In order to optimize the reward from experimentation, it if of great importance that we learn the causal effect of a treatment accurately.
As a result, a stream of literature in online advertising focuses on how to estimate the causal effect of advertisements.
Gordon, Jerath, et al. 2021 reviewed literature in ad effect measurement in digital advertising market.
They pointed out several problems with experiments on ad effect, including small ad effects require large sample size to reach a reasonable statistical power, and experimental cost is extremely high.
Gordon, Zettelmeyer, et al.
2019 empirically compared multiple observational models for causal effects of digital advertising and suggested that common observational approaches fail to accurately measure the true causal effect.
Bumbaca, S. Misra, and
Rossi 2020 proposed an algorithm to efficiently estimate Bayesian hierarchical models for inferences under conditions where researchers have limited number of observations per consumer in online target marketing.
At the framework level, several studies formulated adaptive experimentation, recommendation, and targeted advertising problems as MAB.
Aramayo, Schiappacasse, and Goic 2022 studied complete collections of ads and nonstationary rewards in house ads recommendation problem by using a MAB scheme.
L. Li et al. 2010 applied a contextual bandit framework to model personalized news article recommendation.
Tang et al.
2015 presented implementations of contextual bandit algorithms on a news recommendation problem and an online advertising problem.
Aramayo, Schiappacasse, and
Goic 2022 studied a contextual bandit approach to dynamically exhibit house ads to customers.
Especially, this paper is closely related with CMAB approach, which is a popular branch of models in MAB.
We focus on sequential decision making under the linear contextual bandit framework.
Auer 2002 first introduced the contextual bandit setting through the Linear Reinforcement Learning (LinRel) algorithm with linear value functions.
Subsequently, the contextual framework was improved by Dani, Hayes, and Kakade 2008 and Chu et al.
2011 through Upper Confidence Bound (UCB) type algorithms.
Abbasi-Yadkori, Pál, and Szepesvári 2011 further modified the analysis of the linear bandit problem and improved the regret bound by a logarithmic factor.
More recently, Bastani, Bayati, and Khosravi 2021 studied performance of exploration-free policies in contextual bandit settings and proposed the Greedy-First algorithm which only utilizes observed contexts and rewards to decide whether to follow a greedy algorithm or to explore.
This literature often requires independency between contexts and the random error term, which however cannot be always satisfied in the reality.
We consider a setting which allows the dependency between contexts and error term.
This more general setting is well suited to real-world applications where endogeneity problems happen.
In online settings, data arrives sequentially. This adaptive nature complicates causal inference.
Nie et al. 2018 showed that estimates of arm-specific expected rewards in UCB and TS algorithms are biased downwards.
This downward bias is due to that arms with random upward fluctuation are sampled more, while those arms with downward fluctuation are sampled less (Hadad et al. 2021).
Bibaut et al. 2021 presented that standard estimators no longer follow normal distribution asymptotically so that classic confidence interval fail to provide correct coverage.
To address the inference problem in online settings, H. Chen, Lu, and Song 2021 studied the consistency and asymptotic distribution of the online ordinary least square estimator under epsilon-greedy policy.
We extend their approach to two stage least square estimation in online settings and contribute to the literature by proving the consistency of our estimator which addresses endogeneity problem in online settings.
lattimore2016causal studied the bandit problem on a causal graph and proposed new algorithms to learn good interventions in stochastic online settings.
There are two significant differences between their work and ours.
First, we focus on a single linear model of the features and rewards, while they studied causal graph of features and rewards.
Second, they did not consider the endogeneity problem, which our work mainly contributes to.
This paper is also closely related to the stream of literature in instrumental variable methods for endogeneity problems.
Griliches 1977, Hausman 1983, Angrist and G. Imbens 1995, and X. Chen,
Hong, and Tamer 2005 provided theoretical analysis of instrumental variables in linear models, while
Hansen 1982, Ai and X. Chen 2003, Newey and Powell 2003 and Chernozhukov, G. W. Imbens, and
Newey 2007 studied instrumental variables in nonlinear models.
G. Imbens 2014 reviewed work on instrumental variable methods and discussed applications and restrictions.
Different from the standard data analysis framework in offline settings, the endogeneity problem can be exacerbated during the dynamic interaction between data generation and data analysis in online settings (J. Li, Luo, and
X. Zhang 2021).
This paper builds on existing instrumental variable methods in offline settings and considers the dynamic interaction by utilizing the CMAB framework.
For these reasons, the paper contributes to the literature by combining both CMAB models and instrumental variable methods to improve the estimation of causal effect of ad exposure and hence achieve better performance of decision policies.
Appendix A.3 Further explanation for algorithm
Estimation:
The algorithm maintains four matrices U t , V t , W t , Q t to calculate estimated parameters in each round as described in the pseudo code of Algorithm 2.
If the current time t is after the first round, the algorithm utilizes past-choice-related instrumental variables Z t and past choices X t to estimate Γ 0 through OLS and obtain an estimated X t , which is ˆ X t .
Then, based on ˆ X t and past observations of rewards Y t , the algorithm estimates β 0 by OLS again.
We denote the estimator for β 0 , which is ˆ β t , in the BanditIV algorithm as the BanditIV estimator .
Confidence Sets: Following the Optimisim in the Face of Uncertainty principle (OFU), we need maintain confidence sets for all unknown parameters in the model.
As described in Section 2, we have two unknown parameters, Γ 0 and β 0 .
Thus, we construct two confidence sets C 1 ,s , C 2 ,s , s ∈ { 1 , · · · , T } for the first and the second stage estimation respectively in each round.
The idea for the confidence sets are to make the estimation optimistic with the conditions that "with high probability" the true coefficients are in the confidence sets and we can calculate the confidence sets from the past chosen arms X t , related instrumental variables Z t , and rewards Y t .
Execution: The best arm generated by the algorithm is a sample x t which is related to an instrumental variable z t that can maximize the estimated reward jointly with a pair of optimistic estimates of two-stage coefficients in the confidence sets.
After an arm is chosen, we observe the reward y t as stated in Equation (1). Algorithm
Input: γ z , γ x , { G t } ∞ t =1 , { B t } ∞ t =
Set U
= γ z I ∈ R k × k , V 0 =
∈ R k × d , W
=
γ x I ∈ R d × d , Q 0 =
∈ R d ×
Set C 1 ,s =
{ Γ : || Γ ( i ) − ˆΓ ( i ) s
||
U s ≤ G s } , C 2 ,s = { β : || β −
ˆ β
s || W s ≤ B s } , ∀ s ∈ { 1 , 2 , · · · T } , ∀ i ∈ { 1 , 2 , · · · d }
Nature reveals Z 0 .
We randomly choose z 0 ∈ Z 0 and set Z
=
z
Nature reveals X 0 .
From the set X 0 , we play x 0 which is related with z
and then observe the reward y 0 .
We set X 0 = x 0 and Y
= y 0 for t := 1 , 2 , · · · , T , do
U [Equation]
[Equation]
ˆ β t = W[-
-]
[-+ ˆ X ′ t − 1 ˆ X t − 1 , Q t = Q 
 + ˆ X ′ t − 1 Y t − 1 ˆ β t = W − 1 t Q t-][Equation]
t Nature reveals Z t .
We choose z t = arg max z ∈Z t max Γ ∈ C 1 ,t
Appendix A.4 Confidence intervals
To compute confidence intervals of our estimate of β 0 post running our bandits, we use a rerandomization test similar to that of Bojinov, Simchi-Levi, and Zhao 2020 and Farias et al. 2022.
For the estimate ˆ β , we test the sharp null hypothesis that the β 0 = τ for all t .
The sharp null hypothesis implies that the new outcome is y t + τ ′ ( x Ht − x t ) where x Ht is a newly pulled arm and x t is a pulled arm by our bandit algorithm.
We conduct exact tests by using the known assignment mechanism to simulate new assignment paths.
Algorithm 3 provides the details of how to implement it.
In particular, we propose τ by a downward search method based on the estimate from the bandit algorithm.
Under the sharp null hypothesis [-ofβ 
-]of β 0 = τ , a new arm assignment path 1 : T leads to a sequence of observed outcomes y t + τ ′ ( x Ht − x t ) for t ∈ { 1 , · · · , T } .
To obtain a confidence interval, we invert a sequence of exact hypothesis tests to identify the region outside where the null hypothesis is violated at the prespecified significance level (G. W. Imbens and Rubin 2015, Bojinov, Simchi-Levi, and Zhao 2020).
Algorithm
Test
Require:
Fix N H , total number of samples drawn; Given ˆΓ , X , and Y from the bandit algorithm;
Given a prespecified significance level α for i in 1 : N H do
Sample a new assignment path, x Ht , for t ∈ { 1 , · · · , T } according to the assignment mechanism under the null hypothesis β 0 = τ
Change the sequence of original outcomes y t to y t + ⟨ τ, ( x Ht − x t ) ⟩ for t ∈ { 1 , · · · , T }
Compute ˆ τ [ i ] as the original algorithm does, i.e.
[Equation]
end for
Compute ˆ p = N − 1 H (cid:80) N H [Equation] {| ˆ τ [
[Equation]
] | > | ˆ β |}
Reject the null hypothesis if ˆ p < α
All of these experiments are run on synthetic data constructed as in Section 5.
We run 50 rounds in total.
In each round, we propose 20 hypotheses for β 0 based on the estimated coefficient of our main interest, ˆ β from a pre-run BanditIV or ϵ -greedy BanditIV and construct confidence intervals for all dimensions of β 0 .
For each hypothesis of β 0 , we run the sharp null hypothesis test where we set N H as 200.
We choose the significance level to be α = 0 . 05 .
Under BanditIV, the coverage of the test for each dimension of the estimate is listed as follows: 94%, 96%, 96% ; Under ϵ -greedy
BanditIV, the coverage of the test for each dimension of the estimate is listed as follows: 96%, 92%, 94%.
We can see that, given the significance level as 0 .
05 , the coverage of the test is close to ideal across both BanditIV and ϵ -greedy BanditIV.
Thus, the re-randomization tests and corresponding confidence intervals reported here are adequate for inference of the main treatment effect.
Appendix A.5 Proof of Main Results
Proof for Theorem[-
-]
Proof.
This regret can be decomposed into
[Equation]
Next we will upper bound the first term.
Let (˜Γ t , ˜ β t ) =
arg max Γ ∈ G t − 1 ,β ∈ C t − 1 ⟨ Γ ′ z t , β ⟩ .
By using the two-stage optimism we can decompose our regret as follows,
[Equation]
where the first inequality comes from the Lemma 1, 2 that Γ 0 ∈ C 2 ,t , β 0 ∈ C 1 ,t for all t , the second inequality comes from the definition of z t , the second equality comes from the definition of ˆ x t and some careful decomposition, and the last inequality comes from cauchy-swartz inequality.
Now our goal is to upper bounded these three terms, which mainly comes form the estimation error of Γ 0 in the first stage and the estimation error of β 0 in the second stage.
For the first term, by applying
Cauchy-Schwartz inequality and the standard elliptical potential lemma in Carpentier, Vernade, and
Abbasi-Yadkori 2020, we have
[Equation]
For the second term, by using similar arguments, we have that
[Equation]
where the last inequality again comes from standard ellipse potential lemma.
Finally, for the third term, by the definition of W t , we have ∥ ˜ β t − β
∥ 2 ≤ 1 γ ∥ ˜ β t − β
∥ W t − 1 .
Therefore, by repeating the proof in the second term, we have that,
[Equation]
Therefore, by combing these three terms and the definition of C 1 ,t , C 2 ,t , we can upper bound our regret as
[Equation]
Lemma 1 (Optimistic estimation of β 0 ) .
With high prob 1 − δ/ 2 , for all t ∈ [ T ] ,
[Equation]
where B t = √ γ x || β [-
-]0 || {+
+}
+
[Equation]
Proof.
At any fixed time t , we denote ˆ x ts = Γ ′
t
z s for all s ≤ t , so the ˆ X t defined in the algorithm is a collection of all { ˆ x ts } s ≤ t .
For convenience, we drop the superscript t here.
We also denote e t ∈ R t as a collection of { e s } s ≤ t .
Now we get the closed-form for ˆ β t by ridge 2SLS estimator as follows,
[Equation]
where we denote Z t ( Z ′ t Z t ) − 1 Z t as P Z t .
The third equality comes from the definition of ˆ W t and the upper bound of ˆ X t in Lemma
Therefore, we can write the estimation error of ˆ β compared to β 0 as
[Equation]
Finally, by noticing that e t is 1-subgaussian and the that exp( ⟨ q,
ˆ X ′ t e t ⟩ −
|| q || 2ˆ X ′ t
ˆ X
[Equation]
t) for all q ∈ R d is a supermartingale, according to the Section 20.1 in Lattimore and Szepesvári 2020, we have that, with probability 1 − δ/ 2 T ,
[Equation]
where the last inequality comes from the explicit calculation of W t detailed in Lemma 8.
Lemma 2 (Optimistic estimation of Γ 0 ) .
With high prob 1 − δ/ 2 , for all t ∈ [ T ] and all i ∈ [ d ] , [Equation] i
[Equation]
t where G t = √ γ z || Γ ( i )
||
+ (cid:114) 2 log( 2 Tδ )
+ k log (cid:16) [Table]
Proof.
The proof steps are similar to the previous lemma.
At any fixed time t and dimension i ∈ [ d ] , we denote u it ∈
R t as a collection of { e s } s ≤ t .
We again get the closed-form of ˆΓ t as
[Equation]
And therefore, || ˆΓ
[Equation]
Finally, again by noticing that u it is 1-subgaussian and the that exp( ⟨ q,
ˆ X ′ t u it ⟩ −
|| q || 2ˆ X ′ t ˆ X
[Equation] t) for all q ∈ R d is a supermartingale, we have that, with probability 1 − δ/ 2 T ,
[Equation]
where the last inequality comes from the explicit calculation of U t detailed in Lemma 8 and the union bound over all t ∈
[ T ] .
Lemma 3.
Suppose {F t : t = 1 , · · · , T } is an increasing filtration of σ -fields.
Let { W t : t = 1 , · · · , T } be a sequence of variables such that W t is F t − 1 measurable and | W t | ≤ L w almost surely for all t .
Let { v t : t = 1 , · · · , T } be independent σ v -subgaussian, and v t ⊥ F t − 1 for all t .
Let
S =
{ s 1 , · · · , s |S| } ⊆ { 1 , · · · , T } be an index set where |S| is the number of elements in S .
Then for κ > 0 ,
P ( (cid:88)
[Equation]
The proof of this lemma is provided in Lemma 1 of H. Chen, Lu, and Song 2021.
Lemma 4.
(Dependent OLS Tail Inequality).
For the online decision making model, if all realizations of z t satisfy || z t || ∞ ≤ L z for all t , and ˆΣ = 1 t (cid:80) ts =
z s z ′ s has minimum eigenvalue λ min (ˆΣ) > λ for some λ > 0 almost surely.
Then for any η > 0 ,
[Equation]
Proof.
Based on the proofs provided by H. Chen, Lu, and Song 2021 and Bastani and Bayati 2020, we make some minor changes.
The relation between eigenvalue and l 2 norm of symmetric matrix gives || ˆΣ − 1 || 2 = λ max (ˆΣ − 1 ) =
( λ min (ˆΣ)) − 1 .
Therefore,
[Equation]
Hence, we have
[Equation]
Because we know that v s in the above inequality are i.i.d.
subgaussian and v s ⊥ z s , we can apply
Lemma 3 and have the following
[Equation]
Thus,
[Equation]
Lemma 5.
Let { z t : t = 1 , · · · , T } be a sequence of i.i.d.
k -dimension random vectors such that all realizations of z t satisfy || z t || ∞
≤ L z for all t .
Denote ˆΣ = 1 T (cid:80) Tt
=
z t z ′ t .
If Σ = E [ z t z ′ t ] has minimum eigenvalue λ min (Σ) > λ for some λ > 0 , then
[Equation]
Proof.
This proof is based on the proof of Lemma 3 in H. Chen, Lu, and Song 2021 with minor changes.
First, we have
[Equation]
Then, using the Matrix Chernoff bound,
[Equation]
Lemma 6.
[Equation]
where λ min (ˆΓ) is the smallest magnitude of a singular value of this matrix.
Proof.
By doing Singular Value Decomposition for ˆΓ , we have the following, where the second equality is due to that U is orthonormal.
[Equation]
We assume rank (ˆΓ)
=
min { k, d } = d .
Using matrix form, we have
Σ V ′ (ˆ β − β 0 ) =
  σ · · ·  σ· · · ...
· · · σ d  · · · ...
· · ·    
V (1)(1)
V (2)(1) · · ·
V ( d )(1)
V (1)(2)
V (2)(2) · · ·
V ( d )(2) ...
V (1)( d )
V (2)( d ) · · · V ( d )( d )    
ˆ β
[Table]
) − β (1) ˆ β (2) − β (
[Table]
ˆ β ( d ) − β ( d
[Table]

 =   σ 1 (cid:80) di
=1 V ( i )(1) (ˆ β ( i ) − β ( i ) 0 )
σ 2 (cid:80) di
=1 V ( i )(2) (ˆ β ( i ) − β ( i ) 0 ) ...
σ d (cid:80) di
=1 V ( i )( d ) (ˆ β ( i ) − β ( i ) 0 ) ...

 Hence, [Table]
=
[Table]
=
[Table]
[Equation]
where the last equality is due to that V is orthonormal.
Applying Cauchy–Schwarz inequality, we obtain
[Equation]
[Equation] Combing inequalities (4) and (5), we complete the proof.
Lemma 7.
[Equation]
Proof.
On the event that || ˆΓ − Γ 0 || ≤ η 2 , we can rewrite the equation for each element in ˆΓ using a newly defined matrix A ∈ R k × d : | A ( i )( l ) | ≤ 1 , ∀ i ∈ i ∈ { 1 , 2 , · · · , d } , ∀ l ∈ { 1 , 2 , · · · , k } .
[Equation]
We know that all singular values are non-negative.
Here, we assume the singular values of Γare strictly positive.
We can write the singular values of Γ 0 and ˆΓ as two increasing sequences, 0 < σ (1)Γ ≤ σ (2)Γ ≤ · · · ≤ σ ( d ) Γ and σ (1) ≤ σ (2) ≤ · · · ≤ σ ( d ) , respectively.
Using the min-max principle for singular values, we have
[Equation]
where the first equality is directly derived by applying the min-max theorem.
The second equality is from the two facts.
First, a should be of an order d × 1 to fit the order of Γ 0 .
Second, there are only two unique vectors fitting the constraint ||
a || 2 = 1 in the given space S
and they generate the same objective values[-|| Γ 
-]
|| Γ 0 a || 2 .
Similarly, we can rewrite the smallest singular value of ˆΓ as the following
[Equation]
where the first inequality is derived from the equation (7) and triangle inequality.
The third equality is from the constraint that ||
a || 2 = 1 .
The fourth inequality is from the property of Frobenius norm, which we illustrate as the following
[Equation]
Proposition 2.
(Tail bounds for the online OLS estimators).
In the online decision making model with the ϵ -greedy policy, if Assumptions 1 and 2 are satisfied, and ϵ t is non-increasing, then for any η 1 , η 2 > 0 , any i ∈ { 1 , · · · , d }
[Equation]
Proof for Proposition
Proof.
Following the proof of Proposition 3.1 in H. Chen [Equation] Lu [Equation]
and Song
[Equation]
we provide this proof adapting to our setting.
Denote S t = { [Equation] · · · , t } and define ˆ Σ( I ) =
|I| − 1 (cid:80) s
∈I z s z ′ s for any
I ⊆ S t [Equation] where |I| is the number of element in the set
I and |S t | = t .
We have
[Equation]
Under ϵ -greedy policy, we pull the estimated optimal arm with probability ϵ s at each time point s .
We denote the collection of time points up to time t when the estimated optimal arm is chosen as
R t .
We will first bound the minimum eigenvalue of ˆΣ( R t ) and then use it to infer the bound of the minimum eigenvalue of ˆΣ( S t ) .
We denote the following event,
[Equation]
Applying Lemma 5, if λ min (Σ) > λ , then we have
[Equation]
Meanwhile, we know that
[Equation]
By Weyl’s inequality, on event E ,
[Equation]
Applying Lemma 4, we have
[Equation]
Hence,
[Equation]
The checking step for that |R t | is large enough is exactly the same as that in H. Chen, Lu, and Song 2021.
We complete the proof by combining all the results above,
[Equation]
We can prove the inequality for P ( || ˆΓ ( i ) t − Γ ( i )
|| 1 ≤ η 2 ) by similar steps.
Following from the Proposition 2, we can easily obtain the following corollary.
Corollary 4.
(Consistency of the online OLS estimators).
If Assumptions 1 and 2 are satisfied, ϵ t is non-increasing and tϵ 2 t → ∞ as t → ∞ , then the online OLS estimators ˆ Γ t is a consistent estimator for Γ 0 and ˆ δ t is a consistent estimator for δ 0 .
Proof for Proposition
[-for Proposition 
-]
Proof.
By Proposition 2, we have || ˆΓ − Γ 0 || 1 = max 1 ≤ i ≤ d || ˆΓ ( i ) − Γ ( i )
|| 1 ≤ η 2 with a probability larger than p 2 and ||
ˆ δ − δ 0 || 1 ≤ η 1 with a probability larger than p 1 .
On the events that || ˆΓ − Γ 0 || 1 ≤ ηand
||
ˆ δ −
δ 0 || 1 ≤ η 1 , we can prove the upper bound of ||
ˆ β − β
|| 1 as the following by utilizing triangle inequalities.
[Equation]
Hence,
[Equation]
[Equation] Combining inequality (9), Lemma 6 and Lemma 7, we complete the proof.
Theorem 5.
(Asymptotic normality of the online OLS estimator)
If Assumptions 1 and 2 are satisfied, ϵ t is non-increasing and tϵ 2 t → ∞ as t → ∞ , then
[Equation]
[-Theorem 
-]
Proof.
In this proof, we mainly use martingale central limit theorem.
By definition of the OLS estimator ˆ δ t , we have
[Equation]
We define for 1 ≤ j ≤ t , t ≥ 1 , and any q ∈ R k , the σ -field F tj as F j ,
[Equation]
We have E [ q ′ z
s v s |F s − 1 ] = 0 .
We also notice that { H tj , F tj , 1 ≤ j ≤ t, t ≥ 1 } is a martingale array.
Hence we are going to prove the convergence of H tt .
We first check the Linderberg condition as follows, for each fixed ζ > 0 , t (cid:88) s =
[Equation]
where v s , s = 1 , · · · , t are i.i.d.
from P v .
Notice that v [-
-]21 I { v 21 > [Equation]
|| 22 L 2 z } is dominated by v 21 with
E [ v 21 ] ≤ || β 0 || 22 + 1 .
Also, v 21 I { v 21 > ζ 2 t || q || 22 L 2 z } converges to 0 almost surely when t → ∞ .
Thus, applying dominated convergence theorem, we can get, as t → ∞ ,
[Equation]
Then we are to find the limit of the conditional variance for H tt .
Notice that
[Equation]
and q ′ E [ z s z ′ s ] q ≤ || q
||
L 2 z .
Therefore, t
[Equation]
[Equation]
Applying Martingale Limit Theorem, we have
[Equation]
Now we are to find the limit of 1 t (cid:80) ts =1 q ′
z s z ′ s q for any q ∈ R k .
We know that q ′ z s z ′
s q is F smeasurable for each s and E [
q ′ z s z ′ s q ] ≤ q ′ 11 ′ qL 2 z < ∞ .
Thus, by Theorem 2.19 in Hall and Heyde 2014, we have
[Equation]
From the result in (10) and (12), we have 1 t (cid:80) ts =1 q ′
z s z ′ s q p −→ q ′
[Equation] z s z ′ s d P z ) q .
Applying Lemmain H. Chen, Lu, and Song 2021 and Continuous Mapping Theorem, we obtain
[Equation]
Combining the results in (11) and (13), we complete the proof.
Proof for Theorem 3: Combining the Theorem 5 and Corollary 4, we derive the normality of the
BanditIV estimator by applying Slutsky Theorem directly.
Now, for the consistency of the variance estimator, we want to show that
[Equation]
By rewriting the squared error term in (14), we have
[Equation]
Now we analyze the asymptotic properties for the three terms in (15).
Notice that by Proposition 2, the first term in (15) can be written as [Equation] For the second term, by Proposition 2 and Lemma 11, [Equation]
For the third term
[Equation]
we apply the weak law of large numbers and obtain
[Equation]
Combining the results above and Proposition 2 with continuous mapping theorem, we complete the proof.
Appendix B. Auxiliary Lemmas
Lemma 8.
For each t , we can explicitly upper bound the following terms,
[Equation]
Proof.
For the first inequality,
[Equation]
Then for the second inequality,
[Equation]
Recall that P Z t =
Z ′ t ( Z ′ t Z t ) − 1 Z t , we finish the upper bound of W t .Then
we have
[Equation]
[-d log (cid:18)d ( trace (Γ 0 Γ ′ 
-]
Now because u t is 1-subgaussian
[Equation]
then we have with high probability, u t u ′ t < 2 I .
Then the above is upper bounded by
[Equation]
and similarly
[Equation]
Lemma 9.
Consider a sequence of vectors ( x t )
Tt =1 , x t ∈ R d , and assume that || x t || 2 ≤ a for all t .
Let V t
=
λI
+ (cid:80) ts =1 x s x ′ s for some λ > 0 .
Then, we will have that ||
x t ||
V − 1 t − 1 > b at most d log(1 + a 2 T/λ ) / [-log(
-]log [Equation]
1 + b ) times.
Lemma 9 is directly from lemma 6.2 in Wagenmaker et al. 2021.
Lemma 10.
a 1 , a 2 , · · · ∈ R d is F t adapted and E [ a t |F t − 1 ] = 0 and let L t ∈ R d × d be a predictable sequence such that for any λ ∈ R d ,
[Equation]
Let H t
=
(cid:80) ts =1 a s , J t = (cid:80) ts =1 L t .
[Equation]
[Equation] ) is a super martingale.
Moreover, we have that
[Equation]
Lemma 10 is from the lemma for vector-valued martingale in Kevin’s lecture notes (Proposition 3 in
Section 3.3.5).
Proof for " v ( i ) is ( || β
|| 22 + 1) -subgaussian"
Proof.
[Equation]
where the second equality is due to the independency of distributions of u ( i )( l ) and e ( i ) .
The third inequality is because that u ( i )( l ) and e ( i ) are 1-subgaussian with mean zero. Lemma 11. Suppose {F t : t = 1 , . . .
, T } is an increasing filtration of σ -fields.
Let { ˜ W t : t = 1 , . .
, T } be a sequence of random variables such that ˜ W t is F t − 1 measurable and | ˜ W t | ≤ L ˜ w almost surely for all t .
Let { ˜ e t : t = 1 , . .
, T } be independent σ ˜ e -subgaussian, and ˜ e t ⊥ F t − 1 for all t .
Let ˜ S
= { s 1 , . . .
, s | ˜ S | } ⊆ { 1 , . .
, T } be an index set where | ˜ S | is the number of elements in ˜ S .
Then for κ > 0 ,
[Equation]
[Equation]
Lemma 11 is from Lemma 1 in H. Chen, Lu, and Song 2021.