A R EINFORCEMENT L EARNING F RAMEWORK FOR T IME D EPENDENT C AUSAL E FFECTS E VALUATION IN A/B T ESTING
Anonymous authors
Paper under double-blind review
A BSTRACT
A/B testing, or online experiment is a standard business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries.
The aim of this paper is to introduce a reinforcement learning framework for carrying A/B testing in two-sided marketplace platforms, while characterizing the long-term treatment effects.
Our proposed testing procedure allows for sequential monitoring and online updating.
It is generally applicable to a variety of treatment designs in different industries.
In addition, we systematically investigate the theoretical properties (e.g., size and power) of our testing procedure.
Finally, we apply our framework to both synthetic data and a real-world data example obtained from a technological company to illustrate its advantage over the current practice.
I NTRODUCTION
A/B testing, or online experiment is a business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries (e.g., Google, Amazon, or Facebook).
Most works in the literature focus on the setting, in which observations are independent across time (see e.g. Johari et al., 2015; 2017, and the references therein).
The treatment at a given time can impact future outcomes.
For instance, in a ride-sharing company (e.g., Uber), an order dispatching strategy not only affects its immediate income, but also impacts the spatial distribution of drivers in the future, thus affecting its future income.
In medicine, it usually takes time for drugs to distribute to the site of action.
The independence assumption is thus violated.
The focus of this paper is to test the difference in long-term treatment effects between two products in online experiments.
There are three major challenges as follows.
(i)
The ﬁrst one lies in modelling the temporal dependence between treatments and outcomes.
(ii) Running each experiment takes a considerable time.
The company wishes to terminate the experiment as early as possible in order to save both time and budget.
(iii) Treatments are desired to be allocated in a manner to maximize the cumulative outcomes and to detect the alternative more efﬁciently.
The testing procedure shall allow the treatment to be adaptively assigned.
We summarize our contributions as follows.
First, we introduce a reinforcement learning (RL, see e.g., Sutton & Barto, 2018, for an overview) framework for A/B testing.
In addition to the treatmentoutcome pairs, it is assumed that there is a set of time-varying state confounding variables.
We model the state-treatment-outcome triplet by using the Markov decision process (MDP, see e.g. Puterman, 1994) to characterize the association between treatments and outcomes across time.
Speciﬁcally, at each time point, the decision maker selects a treatment based on the observed state.
The system responds by giving the decision maker a corresponding outcome and moving into a new state in the next time step.
In this way, past treatments will have an indirect inﬂuence on future rewards through its effect on future state variables.
In addition, the long-term treatment effects can be characterized by the value functions (see Section 3.1 for details) that measure the discounted cumulative gain from a given initial state.
Under this framework, it sufﬁces to evaluate the difference between two value functions to compare different treatments.
This addresses the challenge mentioned in (i).
Second, we propose a novel sequential testing procedure for detecting the difference between two value functions.
To the best of our knowledge, this is the ﬁrst work on developing valid sequential tests in the RL framework.
Our proposed test integrates temporal difference learning (see e.g., Precup et al., 2001;
Sutton et al., 2008), the α -spending approach (Lan & DeMets, 1983) and bootstrap (Efron & Tibshirani, 1994) to allow for sequential monitoring and online updating.
It is generally applicable to a variety of treatment designs, including the Markov design, the alternating-time-interval design and the adaptive design (see Section 4.4).
This addresses the challenges in (ii) and (iii).
Third, we systematically investigate the asymptotic properties of our testing procedure.
We show that our test not only maintains the nominal type I error rate, but also has non-negligible powers against local alternatives.
To our knowledge, these results have not been established in RL.
Finally, we introduce a potential outcome framework for MDP.
We state all necessary conditions that guarantee that the value functions are estimable from the observed data.
R ELATED WORK
There is a huge literature on RL such that various algorithms are proposed for an agent to learn an optimal policy and interact with an environment.
Our work is closely related to the literature on offpolicy evaluation, whose objective is to estimate the value of a new policy based on data collected by a different policy.
Popular methods include Thomas et al.
; Jiang & Li (2016); Thomas & Brunskill (2016); Liu et al.
; Farajtabar et al.
; Kallus & Uehara (2019).
Those methods required the treatment assignment probability (propensity score) to be bounded away from 0 and 1.
As such, they are inapplicable to the alternating-time-interval design, which is the treatment allocation strategy in our real data application.
Our work is related to the temporal-difference learning method based on function approximation.
Convergence guarantees of the value function estimators have been derived by Sutton et al.
(2008) under the setting of independent noise and by Bhandari et al.
(2018) for Markovian noise.
However, uncertainty quantiﬁcation of the resulting value function estimators have been less studied.
Such results are critical for carrying out A/B testing.
Luckett et al.
(2019) outlined a procedure for estimating the value under a given policy.
Shi et al.
(2020b) developed a conﬁdence interval for the value.
However, these methods do not allow for sequential monitoring or online updating.
In addition to the literature on RL, our work is also related to a line of research on evaluating timevarying causal effects (see e.g. Robins, 1986; Boruvka et al., 2018; Ning et al., 2019; Rambachan & Shephard, 2019; Viviano & Bradic, 2019; Bojinov & Shephard, 2020).
However, none of the above cited works used an RL framework to characterize treatment effects.
In particular, Bojinov &
Shephard (2020) proposed to use importance sampling (IS) based methods to test the null hypothesis of no (average) temporal causal effects in time series experiments.
Their causal estimand is different from ours since they focused on p lag treatment effects, whereas we consider the long-term effects characterized by the value function.
Moreover, their method requires the propensity score to be bounded away from 0 and 1, and thus it is not valid for our applications.
Furthermore, our work is also related to the literature on sequential analysis (see e.g. Jennison &
Turnbull, 1999, and the references therein), in particular, the α -spending function approach that allocates the total allowable type I error rate at each interim stage according to an error-spending function.
Most test statistics in classical sequential analysis have the canonical joint distribution (see Equation (3.1), Jennison & Turnbull, 1999) and their associated stopping boundary can be recursively updated via numerical integration.
However, in our setup, test statistics no longer have the canonical joint distribution when adaptive design is used.
This is due to the existence of the carryover effects in time.
We discuss this in detail in Appendix C. To resolve this issue, we propose a scalable bootstrap-assisted procedure to determine the stopping boundary (see Section 4.3).
Recently, there is a growing literature on bringing classical sequential analysis to A/B testing.
In particular, Johari et al.
(2015) proposed an always valid test based on the classical mixture sequential probability ratio tests (mSPRT).
Kharitonov et al. (2015) propose modiﬁed versions of the O’Brien & Fleming and MaxSPRT sequential tests. Deng et al. (2016) studied A/B testing under Bayesian framework.
Abhishek & Mannor (2017) developed a bootstrap mSPRT.
These tests cannot detect the carryover effects in time, leading to low statistical power in our setup.
See the toy examples in
Section 4.1 for detailed illustration.
In addition, we note that there is a line of research on bandit/RL with causal graphs (see e.g., Lee & Bareinboim, 2018; 2019).
We remark that the problems considered and the solutions developed in this article are different from these works.
Speciﬁcally, these works considered applying causal inference methods to deal with unmeasured confounders in bandit/RL settings whereas we apply the
RL framework to evaluate time-dependant causal effects.
Finally, we relax several key conditions used in Ertefaie (2014) and Luckett et al.
(2019) that presented a potential outcome framework for MDP (see Section 3.1 for details).
Speciﬁcally, Ertefaie (2014) and Luckett et al.
(2019) imposed the Markov conditions on the observed data rather than the potential outcomes, while assuming that the outcome at time t is a deterministic function of the state variables at time t , t + 1 and the treatment at time t .
P ROBLEM FORMULATION 3. A POTENTIAL OUTCOME FRAMEWORK FOR MDP
For simplicity, we assume that there are only two treatments (actions, products), coded as 0 and 1, respectively.
For any t ≥ 0 , let ¯ a t =
( a 0 , a 1 , · · · , a t ) (cid:62) ∈ { 0 , 1 } t +1 denote a treatment history vector up to time t .
Let S denote the support of state variables and S 0 denote the initial state variable.
We assume S is a compact subset of R d .
For any (¯ a t − 1 , ¯ a t ) , let S ∗ t (¯ a t − 1 ) and Y ∗ t (¯ a t ) be the counterfactual state and counterfactual outcome, respectively, that would occur at time t had the agent followed the treatment history ¯ a t .
The set of potential outcomes up to time t is given by
[Equation]
Let W ∗ = ∪ t ≥ 0 , ¯ a t ∈{ 0 , 1 } t +1 W ∗ t (¯ a t ) be the set of all potential outcomes.
A deterministic policy π is a function that maps the space of state variables to the set of available actions.
For any such π , let ¯ π t denote the treatment history up to time t , assigned according to π .
We use S ∗ t (¯ π t − 1 ) and Y ∗ t (¯ π t ) to denote the associated potential state and outcome that would occur at time t had the agent followed π .
The goodness of a policy π is measured by its value function,
[Equation]
where 0 < γ < 1 is a discounted factor that reﬂects the trade-off between immediate and future outcomes.
Note that our deﬁnition of the value function is slightly different from those in the existing literature (see Sutton & Barto, 2018, for example).
Speciﬁcally, V ( π ; s ) is deﬁned through potential outcomes rather than the observed data.
Similarly, we deﬁne the Q function by
[Equation]
where { ¯ π t ( a ) } t ≥ 0 denotes the treatment history where the initial action equals to a and all other actions are assigned according to π .
In our setup, we focus on two nondynamic policies that assign the same treatment at each time point.
We use their value functions (denote by V (1; · ) and V (0; · ) ) to measure their long-term treatment effects.
Meanwhile, our proposed method is equally applicable to the dynamic policy scenario as well.
See Section B.1 for details.
To quantitatively compare the two policies, we introduce the
Average Treatment Effect (ATE) based on their value functions which relates RL to causal inference.
Deﬁnition.
For a given reference distribution function G , ATE is deﬁned by the integrated difference between two value function, i.e., ATE = (cid:82) s { V (1; s ) − V (0; s ) } G ( ds ) .
The focus of this paper is to test the following hypotheses:
[Equation]
When H 0 holds, the new product is no better than the old one.
3. I DENTIFIABILITY OF ATE
One of the most important question in causal inference is the identiﬁability of causal effects.
In this section, we present sufﬁcient conditions that guarantee the identiﬁability of the value function.
In practice, with the exception of S 0 , the set W ∗ cannot be observed, whereas at time t , we observe the state-action-outcome triplet ( S t , A t , Y t ) .
For any t ≥ 0 , let ¯ A t =
( A 0 , A 1 , · · · , A t ) (cid:62) denote the observed treatment history.
We ﬁrst introduce two conditions that are commonly assumed in multi-stage decision making problems (see e.g. Murphy, 2003; Zhang et al., 2013; Kennedy, 2019).
(CA) Consistency assumption: S t +1 =
S ∗ t +1 ( ¯ A t ) and Y t =
Y ∗ t ( ¯ A t ) for all t ≥ 0 , almost surely.
(SRA)
Sequential randomization: A t is independent of W ∗ given S t and { S j , A j , Y j } 0 ≤ j<t .
The SRA implies that there are no unmeasured confounders and it automatically holds in online experiments, in which the treatment assignment mechanism is pre-speciﬁed.
In SRA, we allows
A t to depend on the observed data history S t , { S j , A j , Y j } 0 ≤ j<t and thus, the treatments can be adaptively chosen.
We next introduce two conditions that are unique to the RL setting.
(MA) Markov assumption: there exists a Markov transition kernel P such that for any t ≥ 0 , ¯ a t ∈ { 0 , 1 } t +1 and S
⊆ R d , we have Pr { S ∗ t +1 (
¯ a t ) ∈ S| W ∗ t (¯ a t ) }
=
P ( S ; a t , S ∗ t (¯ a t − 1 )) .
(CMIA)
Conditional mean independence assumption: there exists a function r such that for any t ≥ 0 , ¯ a t ∈ { 0 , 1 } t +1 , we have E { Y ∗ t (¯ a t ) | S ∗ t (¯ a t − 1 ) , W ∗ t − 1 (¯ a t − 1 ) } =
r ( a t , S ∗ t (¯ a t − 1 )) .
These two conditions are central to the empirical validity of reinforcement learning (RL).
Speciﬁcally, under these two conditions, there exists an optimal policy π ∗ such that V ( π ∗ ; s ) ≥ V ( π ; s ) for any π and s .
We observe that Ertefaie (2014) and Luckett et al.
(2019) imposed the Markov conditions on the observed data rather than the potential outcomes.
When CA and SRA hold, these assumptions are equivalent.
When SRA is violated, their Markov assumptions could be violated as the treatment depends on unobserved confounders and the observed data process is no longer
Markovian.
CMIA requires past treatments to affect Y ∗ t (¯ a t ) only through its impact on S ∗ t (¯ a t − 1 ) .
In other words, the state variables shall be chosen to include those that serve as important mediators between past treatments and current outcomes.
Under MA, CMIA is automatically satisﬁed when
Y ∗ t (¯ a t ) is a deterministic function of ( S ∗ t +1 (¯ a t ) , a t , S ∗ t (¯ a t − 1 )) that measures the system’s status at time t
+ 1 .
The latter condition is commonly imposed in the reinforcement learning literature.
To conclude this section, we derive a version of Bellman equation for the Q function under the potential outcome framework.
Speciﬁcally, for a (cid:48) ∈ { 0 , 1 } , let Q ( a (cid:48) ; · , · ) denote the Q function where treatment a (cid:48) is repeatedly assigned after the initial decision.
Lemma 1 Under MA, CMIA, CA and SRA, for any t ≥ 0 , a (cid:48) ∈ { 0 , 1 } and any function ϕ : S × { 0 , 1 } → R , we have E [ { Q ( a (cid:48) ; A [Equation] , S
[Equation] Sketch of Proof: Under MA, CMIA, CA, SRA, the deﬁned Q-function under the potential outcome framework is the same as that deﬁned on the observed data. Lemma 1 thus follows from the classical Bellman equation (see Equation (4.6) in Sutton & Barto, 2018).
Lemma 1 implies that the Q-function is estimable from the observed data.
Speciﬁcally, an estimating equation can be constructed based on Lemma 1 and the Q-function can be learned by solving this estimating equation.
Note that V ( a, s ) =
Q ( a ; a, s ) and τ 0 is completely determined by the value function V . As a result, τ 0 is estimable from the observed data as well.
We remark that the positivity assumption is not needed in Lemma 1.
Our procedure can thus handle the case where treatments are deterministically assigned, i.e., the behavior policy b is deterministic.
This is due to MA and CMIA that assume the system dynamics are invariant across time.
To elaborate this, note that the discounted value function is completely determined by the transition kernel P and the reward function r .
We remark that these quantities can be consistently estimated under certain conditions (see C1-C3 in Appendix E), regardless of whether b is deterministic or not.  T ESTING PROCEDURE
We ﬁrst introduce a toy example to illustrate the limitations of existing A/B testing methods.
We next present our method and prove its consistency under a variety of different treatment designs.
4. T OY EXAMPLES
Existing A/B testing methods can only detect short-term treatment effects, but fail to identify any long-term effects. To elaborate this, we introduce two examples below. Example 1.
S t = 0 .
5 ε t , Y t =
{+S t + δA t for any t ≥ 1 and S 
+}
= 0 .
5 ε 0 .
Example 2.
S t = 0 .
5 S t − 1 + δA t + 0 .
5 ε t , Y t =
{+S t for any t ≥ 1 and S 
+}
= 0 .
5 ε 0 .
In both examples, the random errors { ε t } t ≥ 0 follow independent standard normal distributions and the parameter δ describes the degree of treatment effects.
Suppose δ > 0 .
Then H 1 holds.
In
Example 1, the observations are independent and there are no carryover effects at all.
In this case, both the existing A/B tests and the proposed test are able to discriminate H 1 from H 0 .
In Example 2, however, treatments have delayed effects on the outcomes.
Speciﬁcally, Y t does not depend on A t , but is affected by A t − 1 through S t .
Existing tests will fail to detect H 1 as the short-term conditional
[Table]
average treatment effects E ( Y t | A t = 1 , S t ) − E
( Y t | A t = 0 , S t )
{+= 
+}
in this example.
As an illustration, we conduct a small experiment by assuming the decision is made once at T = 500 , and report the empirical rejection probability of the classical two-sample t-test that is commonly used in online experiments, a more complicated test based on the double machine learning method (DML,
Chernozhukov et al., 2017) that is widely employed for inferring causal effects, and the proposed test.
It can be seen the competing methods do not have any power under Example 2.
4. A N OVERVIEW OF THE PROPOSAL
First, we estimate τ 0 based on a version of temporal difference learning.
The idea is to apply basis function approximations to solve an estimating equation derived from Lemma 1.
Speciﬁcally, let
Q = { Ψ (cid:62) ( s ) β a (cid:48) ,a : β a (cid:48) ,a ∈ R q } be a large linear approximation space for Q ( a (cid:48) ; a, s ) , where Ψ( · ) is a vector containing q basis functions on S .
The dimension q is allowed to depend on the number of samples T to alleviate the effects of model misspeciﬁcation.
Let us suppose Q ∈ Q for a moment.
By Lemma 1, there exists some β ∗ = ( β ∗(cid:62) 0 , 0 , β ∗(cid:62) 0 , 1 , β ∗(cid:62) 1 , 0 , β ∗(cid:62) 1 , 1 ) (cid:62) such that E [ { Ψ (cid:62) ( S t ) β ∗ a (cid:48)
[Equation]
where I ( · ) denotes the indicator function.
Let ξ ( s, a ) =
{ Ψ (cid:62) ( s ) I ( a = 0) , Ψ (cid:62) ( s ) I ( a = 1) } (cid:62) .
The above equations can be rewritten as E ( Σ t β ∗ ) =
E η t , where Σ t is a block diagonal matrix given by Σ t = (cid:20) [Equation] ξ [Equation]
[Equation] and η t = { ξ ( S t , A t [Equation] (cid:62) Y t , ξ ( S t , A t
[Equation]
} (cid:62) .
Let (cid:98) Σ ( t ) =
t − 1 (cid:80) j<t Σ j and (cid:98) η ( t ) =
t − 1 (cid:80) j<t η j .
It follows that E { (cid:98) Σ ( t ) β ∗ }
= E { (cid:98) η ( t ) } .
This motivates us to estimate β ∗ by (cid:98) β ( t ) =
{ (cid:98) β (cid:62) 0 , 0 ( t ) , (cid:98) β (cid:62) 0 , 1 ( t ) , (cid:98) β (cid:62) 1 , 0 ( t ) , (cid:98) β (cid:62) 1 , 1 ( t ) } (cid:62) =
(cid:98) Σ − 1 ( t ) (cid:98) η
( t ) .
ATE can thus be estimated by the plug-in estimator (cid:98) τ ( t )
=
(cid:82) s Ψ (cid:62) ( s ) { (cid:98) β 1 , 1 ( t ) − (cid:98) β 0 , 0 ( t ) } G ( ds ) .
Second, we use (cid:98) τ ( t ) to construct our test statistic at time t .
We will show √ t { (cid:98) τ ( t ) − τ 0 } is asymptotically normal.
Its variance can be consistently estimated by (cid:98) σ 2 ( t ) =
U (cid:62) (cid:98) Σ − 1 ( t ) (cid:98) Ω ( t ) { (cid:98) Σ − 1 ( t ) } (cid:62) U , as t grows to inﬁnity, where U = {− (cid:82) s ∈ S Ψ( s ) (cid:62) G ( ds ) , 0 (cid:62) q , 0 (cid:62) q , (cid:82) s ∈ S Ψ( s ) (cid:62) G ( ds ) } (cid:62) , 0 q denotes a zero vector of length q , and (cid:98) Ω ( t ) corresponds to some consistent covariance estimator of η t based on the data observed at time t (see equation 3 for the explicit form).
This yields our test statistic √ t (cid:98) τ ( t ) / (cid:98) σ ( t ) , at time t .
Third, we integrate the α -spending approach with bootstrap to sequentially implement our test (see
Section 4.3).
The idea is to generate bootstrap samples that mimic the distribution of our test statistics, to specify the stopping boundary at each interim stage.
Suppose that the interim analyses are conducted at time points T 1 < · · · < T K = T .
To simplify the presentation, for each 1 ≤ k < K , we assume T k /T
→ c k for some constants 0 < c 1 < · · · < c K − 1 < 1 .
To better understand our algorithm, we next investigate the limiting distribution of our test statistics at these interim stages.
4. S EQUENTIAL MONITORING AND ONLINE UPDATING
Let { Z 1 , · · · , Z K } denote the sequence of our test statistics, where Z k = √ T k (cid:98) τ ( T k ) / (cid:98) σ ( T k ) .
To sequentially monitor our test, we need to specify the stopping boundary { b k } 1 ≤ k ≤ K such that the experiment is terminated and H 0 is rejected when Z k > b k for some k .
First, we use the α spending function approach to guarantee the validity of our test.
It requires to specify a monotonically increasing function α ( · ) that satisﬁes α (0)
=
and α ( T ) =
α .
Some popular choices of the α spending function include
[Equation]
where Φ( · ) denotes the normal cumulative distribution function.
Adopting the α spending approach, we require b k ’s to satisfy
[Equation]
As commented in the introduction, the numerical integration method is not applicable to determine the stopping boundary.
Our method is built upon the wild bootstrap (Wu et al., 1986).
The idea is to generate bootstrap samples that have asymptotically the same joint distribution as the test statistics.
However, we note that directly applying the wild bootstrap algorithms is time consuming.
See Section C for details.
To facilitate the computation, we present a scalable bootstrap algorithm to determine { b k } k .
Let { e k } k be a sequence of i.i.d N (0 , I 4 q ) random vectors, where I J stands for a J × J identity matrix for any J .
Let (cid:98) Ω
( T 0 ) be an 4 q × 4 q zero matrix.
At the k -th stage, we compute
[Equation]
A key observation is that, conditional on the observed dataset, the covariance of (cid:98) Z ∗ k 1 and (cid:98) Z ∗ k 2 equals that of Z k 1 and Z k 2 .
See Theorem 3 for details.
In addition, the limiting distributions of { Z k } k and { Z ∗ k } k are multivariate normal.
As such, the joint distribution of { Z k } k can be well approximated by that of { Z ∗ k } k conditional on the data.
This forms the basis of our bootstrap algorithm.
By the requirement on { b k } k in 2, we obtain Pr (max 1 ≤ j<k
( Z j − b j ) ≤ 0 , Z k > b k ) =
α ( T k ) − α
( T k − 1 )
+ o (1) .
To implement our test, we recursively calculate the threshold (cid:98) b k as follows,
[Equation]
where Pr ∗ denotes the conditional probability given on the data, and reject H 0 when Z ∗ k > (cid:98) b k for some k .
In practice, the left-hand-side can be approximated via Monte carlo simulations.
4. C ONSISTENCY UNDER DIFFERENT TREATMENT DESIGNS
We consider three treatment allocation designs that can be handled by our procedure as follows:
D1 .
Markov design: Pr ( A t = 1 | S t , { S j , A j , Y j } 0 ≤ j<t ) =
b (0) ( S t ) for some function b (0) ( · ) uniformly bounded away from 0 and 1 . D2 .
Alternating-time-interval design: A 2 j = 0 , A 2 j +1 = 1 for all j ≥ 0 . D3 . Adaptive design (e.g., (cid:15) -greedy): For T k ≤ t < T k +1 for some k ≥ 0 , Pr ( A t = 1 | S t , { S j , A j , Y j } 0 ≤ j<t ) =
b ( k ) ( S t ) for some b ( k ) ( · ) that depends on { S j , A j , Y j } 0 ≤ j<T k . Here, D2 is a deterministic design and is widely used in industry (see our real data example). D1 and D3 are random designs.
D1 is commonly assumed in the literature on off-policy evaluation (see e.g., Jiang & Li, 2016). D3 is widely employed in the contextual bandit setting to balance the trade-off between exploration and exploitation. These three settings cover a variety of scenarios in practice.
Theorem 1 (Type-I error) Suppose α ( · ) is continuous, C1-C3 (see Appendix E) hold and q = o ( √ T/ log T ) . Then Pr ( (cid:83) kj =1 { Z j > (cid:98) b j } ) ≤ α ( T k ) + o (1) , for all 1 ≤ k ≤ K under H 0 . Sketch of Proof: We consider the case where τ 0 = 0 only.
The general case is proven in Section F.3. As discussed in Section 4.3, the conditional distribution of { Z ∗ k } k given the data is equivalent as the distribution of { Z k } k . Since { (cid:98) b k } k is a continuous function of { Z ∗ k } k , it follows from the continuous mapping theorem that { (cid:98) b k } k are consistent. The proof is hence completed.
Theorem 1 implies that the type-I error rate of the proposed test is well controlled.
When ATE = 0 , the equality in Theorem 1 holds. The rejection probability achieves the nominal level under H 0 .
Theorem 2 (Power) Under the conditions of Theorem 1, assume τ 0 (cid:29) T − 1 / 2 , then Pr ( Z 1 > (cid:98) b 1 ) → 1 .
Assume τ 0 =
T − 1 / 2 h for some h > 0 .
Then lim T →∞
[ Pr ( ∪ kj =1 { Z j > (cid:98) b j } ) − α ( T k )] > 0 .
Sketch of Proof: Under H 1 , similar to Theorem 1, we have Pr ( ∪ kj =1 { Z j − (cid:112) T j τ 0 / (cid:98) σ ( T j ) > (cid:98) b j } = α ( T k ) + o (1) .
The assertion follows by that Z j is stochastically larger than Z j − (cid:112) T j τ 0 for all j .
The second assertion in Theorem 2 implies that our test has non-negligible powers against local alternatives converging to H 0 at the T − 1 / 2 rate.
When the signal decays to zero faster than this rate, our test is not able to detect H 1 .
When the signal decays at a slower rate, the power of our test approaches 1.
Combining Theorems 1 and 2 yields the consistency of our test.
Finally, it is worth mentioning that our test can be online updated as batches of observations arrive at the end of each interim stage.
We summarize our procedure in Algorithm 1 (see Appendix A).
Its time complexity is dominated by O ( Bq 3 + Tq 2 ) . [Figure]
N UMERICAL STUDIES 5. S YNTHETIC DATA
Simulated data of states and rewards was generated as follows,
S 1 ,t = (2 A t − 1 − 1) S 1 , ( t − 1) / 2 + S 2 , ( t − 1) / 4 + δA t − 1 + ε 1 ,t ,
S 2 ,t = (2 A t − 1 − 1) S 2 , ( t − 1) / 2 + S 1 , ( t − 1) / 4 + δA t − 1 + ε 2 ,t , Y t = 1 + ( S 1 ,t + S 2 ,t ) / 2 + ε 3 ,t , where the random errors { ε j,t } j =1 , 2 , 0 ≤ t ≤ T are i.i.d N (0 , 0 .
5 2 ) and { ε 3 ,t } 0 ≤ t ≤ T are i.i.d N (0 , 0 .
The initial states S 1 , 0 and S 2 , 0 are independent N (0 , 0 .
5 2 ) as well.
Let S t = ( S 1 ,t , S 2 ,t ) (cid:62) denote the state at time t .
Under this model, treatments have delayed effects on the outcomes, as in Example 2.
The parameter δ characterizes the degree of such carryover effects.
When δ = 0 , τ [-
-]0 = 0 and Hholds.
When δ > 0 , H 1 holds.
Moreover, τ 0 increases as δ increases.
We set K =
and ( T 1 , T 2 , T 3 , T 4 , T 5 ) =
The discounted factor γ is set to 0 .
6 and G is chosen as the initial state distribution.
We consider three behavior policies, according to the designs D1-D3, respectively.
For the behavior policy in D1, we set b (0) ( s ) = 0 .
5 for any s ∈ S .
For the behavior policy in D3, we use an (cid:15) -greedy policy and set b ( k ) ( s ) =
(cid:15)/ 2 + (1 − (cid:15) )
I (Ψ( s ) (cid:62) ( (cid:98) β 1 , 1 ( T k ) − (cid:98) β 0 ,
( T k )) > 0) , with (cid:15) = 0 .
1 , for any k ≥ 1 and s ∈ S .
For each design, we further consider ﬁve choices of δ , corresponding to 0 , 0 . 05 , 0 . 1 , 0 .
15 and 0 . 2 .
The signiﬁcance level α is set to 0 .
in all cases.
To implement our test, we choose two α -spending functions, corresponding to α 1 ( · ) and α 2 ( · ) given in equation 1.
The hyperparameter θ in α 2 ( · ) is set to 3 .
The number of bootstrap sample is set to 1000 .
In addition, we consider the following polynomial basis function, Ψ( s ) =
Ψ( s 1 , s 2 ) =
(1 , s 1 , s 21 , · · · , s J 1 , s 2 , s 22 , · · · , s J 2 ) (cid:62) , with J = 4 .
We also tried some other values of J by setting J to 3 and 5.
Results are reported in Figure 6 (see
Appendix G).
It can be seen that the resulting tests is not sensitive to the choice of J .
All experiments run on a macbook pro with a dual-core 2.7 GHz processor.
Implementing a single test takes one second.
Figures 1 (a) and 5 (a) (see Appendix G) depict the empirical rejection probabilities of our test statistics at different interim stages under H 0 and H 1 with different combinations of δ , α ( · ) and the designs.
These rejection probabilities are aggregated over 500 simulations.
We also plot α 1 ( · ) and α 2 ( · ) under H 0 .
Based on the results, it can be seen that under H 0 , the type-I error rate of our test is well-controlled and close to the nominal level at each interim stage.
Under
H 1 , the power of our test increases as δ increases, showing the consistency of our test procedure.
To further evaluate our method, we compare it with the classical two-sample t-test and the sequential test developed by Kharitonov et al.
To apply the t-test, for each T k , we apply the t-test to the data { A t , Y t } 0 ≤ t ≤ T k and plot the corresponding empirical rejection probabilities in Figures 1(b) and 5(b) (Appendix G). Results for Kharitonov et al. (2015)’s
test are reported in Figure 4 (Appendix G). Both competing methods fail to detect any carryover effects and do not have any power.
We next explain why several other methods mentioned in the introduction cannot be used for comparison.
First, a lot of causal effects evaluation methods did not consider early termination.
Consequently, they are unsuitable to apply in our numerical studies.
Second, standard temporal difference learning method did not study the asymptotic distribution of the resulting value estimators.
These results are critical for carrying out A/B testing.
Finally, many methods proposed to use inverse propensity-score weighting.
These methods are not valid for the alternating-time-interval design.
5. R EAL DATA APPLICATION
We apply the proposed test to a real dataset from a ride-sharing platform.
Order dispatching is one of the most critical problems in online ride-hailing platforms to adapt the operation and management strategy to the dynamics in demand and supply.
The purpose of this study is to compare the performance of a newly developed strategy with a standard control strategy used in the platform.
The new strategy is expected to reduce the answer time of passengers and increase drivers income.
For a given order, the new strategy will dispatch it to a nearby driver that has not yet ﬁnished their previous ride request, but almost.
In comparison, the standard control assigns orders to drivers that have completed their ride requests.
The experiment is conducted at a given city from December 3rd to December 16th.
Dispatch strategies are executed based on alternating half-hourly time intervals.
We also apply our test to a data from an A/A experiment (which compares the baseline strategy against itself), conducted from
November 12th to November 25th.
We expect that our test will not reject H 0 when applied to the data from the A/A experiment, since the two strategies used are essentially the same.
Both experiments last for two weeks.
Thirty-minutes is deﬁned as one time unit.
We set T k = {+
+} ( k + 6) for k = 1 , . . .
That is, the ﬁrst interim analysis is performed at the end of the ﬁrst week, followed by seven more at the end of each day during the second week.
We choose the overall drivers’ income in each time unit as the response.
The new strategy is expected to reduce the answer time of passengers and increase drivers’ income.
Three time-varying variables are used to construct the state.
The ﬁrst two correspond to the number of requests (demand) and drivers’ online time (supply) during each 30-minutes time interval.
These factors are known to have large impact on drivers’ income.
The last one is the supply and demand equilibrium metric.
This variable characterizes the degree that supply meets the demand and serves as an important mediator between past treatments and future outcomes.
To implement our test, we set γ = 0 .
6 , B = 1000 and use a fourth-degree polynomial basis for Ψ( · ) , as in simulations.
We use α 1 ( · ) as the spending function for interim analysis and set α = 0 . 05 .
The test statistic and its corresponding rejection boundary at each interim stage are plotted in Figure 2.
It can be seen that our test is able to conclude, at the end of the 12th day, that the new order dispatch strategy can signiﬁcantly increase drivers’ income, and meet more order requests.
In addition, based on the dataset from the A/B experiment, we found that the new strategy reduces the answer time of orders by 2%, leading to almost 2% increment of drivers income.
When applied to the data from the
A/A experiment, we fail to reject H 0 , as expected.
For comparison, we also apply the two-sample ttest to the data collected from the A/B experiment.
The corresponding p-value is 0.18.
This result is consistent with our ﬁndings. Speciﬁcally, the treatment effect at a given time affects the distribution of drivers in the future, inducing interference in time. As shown in the toy example (see Section 4.1), the t-test cannot detect such carryover effects, leading to a low power. Our procedure, according to
Theorem 2, has enough powers to discriminate H 1 from H 0 .
A M ORE ON THE ALGORITHM
A pseudo algorithm summarizing our procedure is given in Algorithm 1.
We next introduce some notations.
The matrix (cid:98) Ω ( t ) is deﬁned by (cid:98) Ω ( t ) = 1 t t − 1 (cid:88)
j = (cid:18)
[-here 
 q denotes a zero vector of size q . The matrix-]ξ j (cid:98)
[Equation]
[Equation]
where (cid:98) ε j, 0 and (cid:98)
ε j, 1 are the temporal difference errors deﬁned in Algorithm 1.
B E XTENSIONS B. E XTENSIONS TO DYNAMIC POLICIES
In this paper, we focus on comparing the long-term treatment effects between two nondynamic policies.
The proposed method can be easily extended to handle dynamic policies as well.
Speciﬁcally, consider two time-homogeneous policies π 1 and π 2 where each π j ( s ) measures
the treatment assignment probability Pr
A t = 1 | S t = s ) .
Note that the integrated value difference function τ 0 can be represented by
[Equation]
The Q-estimators can be similarly computed via temporal difference learning.
More speciﬁcally, for a given policy π , let
[Equation]
[Figure] be the Q-estimator given the data { ( S j , A j , Y j ) } j<t where (cid:98)
Σ π ( t ) =
t − 1 (cid:80) j<t Σ j where Σ j is deﬁned by
[Equation]
We can plug-in the Q-estimator in equation 4 to estimate τ 0 .
The corresponding variance estimator and the resulting test statistic can be similarly derived.
A bootstrap procedure can be similarly developed as in Section 4.3 for sequential testing.
We omit the details for brevity.
B. E XTENSIONS TO OTHER NONPARAMETRIC ESTIMATORS
In addition to temporal difference learning, other existing OPE methods could be potentially coupled with the proposed bootstrap procedure for online sequential testing. We use the double reinforcement learning method (DRL, Kallus & Uehara, 2019) as an example. First, we remark that DRL requires the system to be ergodic and use an inverse propensity-score weighted method to construct the value estimator.
As such, it might not be applicable to the alternating-time-interval design and the adaptive design.
Second, in the Markov design, it could be coupled with our bootstrap procedure for sequential testing.
We compare such a procedure with our proposed method using the simulation setting in
Section 5.1, and report the rejection probabilities in Figure 3.
It can be seen that DRL-based test has some inﬂated type-I errors under H 0 and is less powerful than our procedure under H 1 .
We next outline the procedure.
Speciﬁcally, at the k th interim stage, we compute the test statistic
[Equation]
where
[Equation] Q k and (cid:98) ω k denote the estimated Q- and marginal density ratio functions based on the data collected at the k th stage, and [Equation]
The bootstrapped sample can be constructed as
[Equation]
Then similar to Algorithm 1, we can decide whether to reject H 0 or not based on the test statistic Z k and the bootstrap samples. This algorithm can be implemented online provided that (cid:98) Q and (cid:98) ω can be computed online.
We ﬁrst provide an example to show that our test statistics do not have the canonical joint distribution. This motivates us to propose a wild bootstrap algorithm. We next present some details on the bootstrap algorithm.
Let { Z k } k be the sequence of test statistics conducted at each interim stage. These test statistics are said to have canonical joint distribution with information levels {I k } k for the parameter θ if: (i)
[Equation]
Z 1 , Z 2 , · , Z K ) (cid:62) is asymptotically normal,
[Equation]
ii
[Equation]
See also Equation (3.1) in Jennison & Turnbull (1999). Unlike the settings where observations are independent across time, (iii) is likely to be violated in our setup when adaptive design is used. This is due to the existence of carryover effects in time. Speciﬁcally, when treatment effects are adaptively generated, the behavior policy at difference stages are likely to vary.
Due to the carryover effects in time, the state vectors at difference stages have different distribution functions. According to Part 3 of the proof of Theorem 3, we have for any k 1 ≤ k 2 that
[Equation]
The matrices Σ ( k ) and Ω ( k ) depend the distributions of state vectors and are likely to differ for different k . Consequently, the second term η k 1 ,k 2 on the right-hand-side depends on both k 1 and k 2 .
As such, (iii) is violated. The idea of our bootstrap algorithm is to generate bootstrap samples { (cid:98) Z MB ( t ) } t that have asymptotically the same joint distribution as {√ t (cid:98) σ − 1 ( t )( (cid:98)
τ ( t ) − τ 0 ) } t .
Speciﬁcally, let { ζ t } t ≥ 0 be a sequence of i.i.d.
random variables independent of the observed data.
Deﬁne
[Equation]
where (cid:98) ε t,
a is the temporal difference error deﬁned in Algorithm 1.
Based on (cid:98) β MB ( t ) , one can deﬁne the bootstrap sample
[Equation]
We remark that although the wild bootstrap method is developed under the i.i.d.
settings, it is valid under our setup as well.
This is due to that under CMIA, (cid:98) β ( t ) − β ∗ forms a martingale sequence with respect to the ﬁltration { ( S j , A j , Y j ) :
j < t } .
This guarantees that the covariance matrices of (cid:98) β MB ( t ) and (cid:98) β ( t ) are asymptotically equivalent.
As such, the bootstrap approximation is valid.
However, calculating (cid:98) β MB ( T k ) requires O ( T k ) operations.
The time complexity of the resulting bootstrap algorithm is O
( BT k ) up to the k -th interim stage, where B is the total number of bootstrap samples.
This can be time consuming when { T k − T k − 1 } Kk =1 are large.
D M ORE ON THE DESIGNS
In D3, we require b ( k ) to be strictly bounded between 0 and 1.
Suppose an (cid:15) -greedy policy is used, i.e. b ( k ) ( s ) =
(cid:15)/ 2 + (1 − (cid:15) ) (cid:98) π ( k ) ( s ) , where (cid:98) π
( k ) denotes some estimated optimal policy.
It follows that (cid:15)/ 2 ≤ b ( k ) ( s ) ≤ 1 − (cid:15)/ 2 for any s .
Such a requirement is automatically satisﬁed.
For any behaviour policy b in D1-D3, deﬁne S ∗ t (¯ b t − 1 ) and Y ∗ t (¯ b t ) as the potential outcomes at time t , where ¯ b t denotes the action history assigned according to b .
When b is a random policy as in D1 or D3, deﬁnitions of these potential outcomes are more complicated than those under a deterministic policy (see Luckett et al., 2019).
When b is a stationary policy, it follows from MA that { S ∗ t +1 (¯ b t ) }
t ≥− 1 forms a time-homogeneous Markov chain.
When b follows the alternatingtime-interval design, both { S ∗ 2 t (¯ b 2 t − 1 ) } t ≥ 0 and { S ∗ 2 t +1 (¯ b 2 t ) } t ≥ 0 form time-homogeneous Markov chains.
We next show that ( Z 1 , · · · , Z K ) is asymptotically multivariate normal and provide a consistent covariance estimator.
Theorem 3 (Limiting distributions) Assume C1-C3 hold.
Assume all immediate rewards are uniformly bounded variables, the density function of S 0 is uniformly bounded on S and q satisﬁes q = o ( √ T/ log T ) .
Then under either D1, D2 or D3, we have
• { Z k } 1 ≤ k ≤ K are jointly asymptotically normal; • their asymptotic means are non-positive under H 0 ; • their covariance matrix can be consistently estimated by some (cid:98) Ξ , whose ( k 1 , k 2 ) -th element (cid:98) Ξ k 1 ,k 2 equals (cid:112) T k 1 /T k 2 U (cid:62) (cid:98) Σ − 1 ( T k 1 ) (cid:98) Ω
( T k 1 ) { (cid:98) Σ − 1 ( T k 2 ) } (cid:62) U / { (cid:98) σ ( T k 1 ) (cid:98) σ ( T k 2 ) } .
E T ECHNICAL CONDITIONS
To simplify the presentation, we assume all state variables are continuous.
The immediate reward and the density function of S 0 are bounded.
E. C ONDITION C
C1 Suppose (i) holds.
Assume (ii) holds under D1, (iii) holds under D2 and (ii), (iv) hold under D3.
(i)
The transition kernel P is absolutely continuous and satisﬁes P ( ds ; a, s (cid:48) = p ( s a s
(cid:48) ) ds for some transition density function p
In addition, assume p is uniformly bounded away from 0 and ∞
(ii)
The Markov chain { S ∗ t (¯ b (0) t − 1 ) } t ≥ 0 formed under the behaviour policy b (0) is geometrically ergodic, i e. there exists some function M on S , some constant 0 ≤ ρ < 1 and some probability density function Π such that (cid:82) s ∈ S M ( s )Π( ds ) < + ∞ and
[Equation]
where (cid:107) · (cid:107) TV denotes the total variation norm
(iii)
The Markov chains { S ∗ 2 t (¯ b 2 t ) } t ≥ 0 and { S ∗ 2 t +1 (¯ b 2 t +1 ) } t ≥ 0 are geometrically ergodic.
(iv) For any k = 1 , · · · , K − 1 , the following events occur with probability tending to 1 : the Markov chain { S ∗ t (¯ b ( k ) t − 1 ) } t ≥ 0 is geometrically ergodic; sup s ∈ S | b ( k ) ( s ) − b ∗ ( s )
| P → 0 for some b ∗ ( · ) ; the stationary distribution of { S ∗ t (¯ b ( k ) t − 1 ) } t ≥ 0 will converge to some Π ∗ in total variation.
Remark : By C1(ii), Π is the stationary distribution of { S ∗ t (¯ b (0) t − 1 ) } t ≥ 0 .
It follows that
[Equation]
for any S ⊆ S .
By C1(i), we obtain Π
[Equation]
) = [Equation] This implies that µ ( · ) is the density function of Π .
Since p is uniformly bounded away from 0 and ∞ , so is µ .
Under C1(iv), for any k ∈ { 1 , · · · , K − 1 } , there exist
some M ( k ) ( · ) , Π ( k ) ( · ) and ρ ( k ) that satisfy (cid:82) s ∈ S M ( k ) ( s )Π ( k ) ( ds ) <
+ ∞ and (cid:13)(cid:13)(cid:13)
Pr ( S ∗ t ( ¯ b ( k ) t − 1 ) ∈ S [Equation] [Equation]
[Equation]
( k
[Equation]
S )
[Equation]
TV [Equation] ( k [Equation]
( s
[Equation]
{ ρ ( k ) } t
[Equation]
with probability tending to 1 .
Since b ( k ) is a function of the observe data history, so are M ( k ) ( · ) , Π ( k ) ( · ) and ρ ( k ) .
Suppose an (cid:15) -greedy policy is used, i.e. b ( k ) ( s )
=
(cid:15)/ 2 + (1 − (cid:15) ) (cid:98) π ( k ) ( s ) where (cid:98) π
( k ) denotes some estimated optimal policy.
Then the condition sup s ∈ S | b ( k ) ( s ) − b ∗ ( s ) | P → 0 requires (cid:98) π ( k ) to converge.
The total variation distance between the one-step transition kernel under ¯ b ( k ) and that under b ∗ can be bounded by
[Equation]
and converges to zero in probability.
When the markov chain { S ∗ t (¯ b ( k ) t − 1 ) } t ≥ 0 is uniformly ergodic, it follows from Theorems 2 and 3 of Rabta & A¨ıssani (2018) that (cid:107) Π ( k ) − Π ∗ (cid:107) TV → 0 where Π ∗ corresponds to the stationary distribution of { S ∗ t (¯ b ∗ t − 1 ) } .
The last condition in C1(iv) is thus satisﬁed.
E. C ONDITION C
C2(i)
Assume there exists some β ∗ such that
[Equation]
(ii) Assume there exists some constant ¯ c ∗ ≥ 1 such that
[Table]
and sup s (cid:107) Ψ( s ) (cid:107) {+
+}
= O ( √ q ) .
(iii) Assume lim inf q (cid:107) (cid:82) s
∈ S Ψ
( s ) G ( ds ) (cid:107) 2 > 0 .
Remark :
For any a, a (cid:48) ∈ { 0 , 1 } , suppose Q ( a (cid:48) ; a, s ) is p -smooth as a function of s (see e.g. Stone, 1982, for the deﬁnition of p -smoothness).
When tensor product B-splines or wavelet basis functions (see Section 6 of Chen & Christensen, 2015, for an overview of these bases) are used for Ψ( · ) , we have sup
[Equation]
under certain mild conditions.
See Section 2.2 of Huang (1998) for details.
It follows that Condition
C2(i) automatically holds when the number of basis functions q satisﬁes q (cid:29) T d/ (2 p ) .
Condition C2(ii) is satisﬁed when tensor product B-splines or wavelet basis is used.
For B-spline basis, the assertion in equation 8 follows from the arguments used in the proof of Theorem 3.3,
Burman & Chen (1989).
For wavelet basis, the assertion in equation 8 follows from the arguments used in the proof of Theorem 5.1, Chen & Christensen (2015).
For both bases, the number of nonzero elements in Ψ( · ) is bounded by some constant.
Moreover, each basis function is uniformly bounded by O ( √ q ) .
The condition sup s (cid:107) Ψ( s ) (cid:107)
=
O ( √ q ) thus holds.
Condition C2(iii) automatically holds for tensor product B-splines basis.
Notice that 1 (cid:62) Ψ( s )
=
q 1 /for any s ∈ S where 1 denotes a vector of ones.
It follows from Cauchy-Schwarz inequality that
[Equation]
C2(iii) is thus satisﬁed.
E. C ONDITION C
Let ε ∗ ( a (cid:48) , a ) =
Y ∗
( a ) + γQ ( a (cid:48) ; a (cid:48) , S ∗ 1 ( a ))
− Q ( a (cid:48) ; a, S 0 ) .
C3 Assume inf q inf a (cid:48) ,a ∈{ 0 , 1 } ,s ∈ S Var { ε ∗ ( a (cid:48) , a ) | S
= s } > 0 and sup q sup a ∈{ 0 , 1 } ,s ∈ S ρ ε ( a, s ) <
where
[Equation]
Here, ρ ε corresponds to the partial correlation of ε ∗ (0 , a ) and ε ∗ (1 , a ) given S 0 .
F T ECHNICAL PROOFS F. P ROOF OF L EMMA
To prove Lemma 1, we state the following lemma.
Lemma 2 Under MA and CMIA, Q ( a (cid:48) ; a, s ) =
r ( a, s ) + γ (cid:82) s (cid:48) Q ( a (cid:48) ; a (cid:48) , s (cid:48) )
P ( ds (cid:48) ; a, s ) for any ( s, a ) .
Proof of Lemma 2: For any a, a (cid:48) ∈ { 0 , 1 } , deﬁne the potential outcome Y ∗ t ( a (cid:48) , a ) and S ∗ t ( a (cid:48) , a ) as the reward and state variables that would occur at time t had the agent assigned Treatment a at the initial time point and Treatment a (cid:48) afterwards.
Let P ta (cid:48)
( S , a, s ) =
Pr { S ∗ t ( a (cid:48) , a ) ∈ S | S
= s } for any S ⊆ S , a, a (cid:48) ∈ { 0 , 1 } , s ∈ S and t ≥ 0 .
We break the proof into two parts.
In Part 1, we show Lemma 2 holds when the following is satisﬁed: [Equation] P ta
[Equation]
In Part2, we show equation 9 holds.
Part 1: Under CMIA, we have
[Equation]
It follows that
[Equation]
Similar to equation 10, we can show
[Equation]
and hence (cid:88)
[Equation]
By equation [Equation] the conditional distribution of S ∗ t +1 ( a (cid:48) , a ) given S ∗ 1 ( a ) =
s and S 0 are the same as the conditional distribution of S ∗ t ( a (cid:48) , a ) given S
= s .
It follows that from equation 11 that (cid:88)
[Equation]
This together with the deﬁnition of Q function and CMIA yields
[Equation]
[Equation]
Under MA, we have
[Equation]
Combining this together with equation 12 yields the desired result.
Part 2: We use induction to prove equation 9.
When t = 0 , it trivially holds.
Suppose equation 9 holds for t = k .
In the following, we show equation 9 holds for t = k + 1 .
Under MA, we have
[Equation]
Since we have shown equation 9 holds for t =
k , it follows that
[Equation]
Similarly
[Equation]
we can show
[Equation]
The proof is hence completed.
Proof of Lemma 1 : By CA, it is equivalent to show
[Equation]
Let S 0 denote the support of S 0 .
For any s 0 ∈ S 0 , it sufﬁces to show
[Equation]
This is equivalent to show
[Equation]
for any s 0 ∈ S 0 , a 0 ∈ { 0 , 1 } .
Let A
( s 0 )
= { a ∈ { 0 , 1 } : Pr
( A 0 = a | S
= s ) > 0 } .
It sufﬁces to show for any s 0 ∈ S 0 , a 0 ∈ A [Equation]
[Equation]
or equivalently,
[Equation]
Let ¯ s j
= ( s 0 , s 1 , · · · , s j ) (cid:62) , ¯ y j = ( y 0 , y 1 , · · · , y j ) (cid:62) , ¯ S
j
= ( S 0 , S 1 , · · · , S j ) (cid:62) and ¯ Y j
= ( Y 0 , Y 1 , · · · , Y j ) (cid:62) .
We can recursively deﬁne the sets Y j (¯ s j , ¯ a j , ¯ y j − 1 ) , S j +1 (¯ s j , ¯ a j , ¯ y j ) ,
A j +1 (¯ s j +1 , ¯ a j , ¯ y j ) to be the supports of Y j , S j +1 , A j +1 conditional on ( ¯ S
j = ¯ s j , ¯ A j = ¯ a j , ¯ Y j − 1 =
¯ y j − 1 ) , ( ¯ S j = ¯ s j , ¯ A j = ¯ a j , ¯ Y j = ¯ y j ) , ( ¯ S j +
= ¯ s j +1 , ¯ A j = ¯ a j , ¯ Y j = ¯ y j ) respectively, for j ≥ 0 .
Similar to equation 13, it sufﬁces to show E
[Equation]
, for any s 0 ∈ S 0 , a 0 ∈ A 0 ( s 0 ) , y 0 ∈ Y 0 ( s 0 , a 0 ) , · · · , s t ∈ S t (¯ s t − 1 , ¯ a t − 1 , ¯ y t − 1 ) , a t ∈
A t (¯ s t , ¯ a t − 1 , ¯ y t − 1 ) .
This is equivalent to show
E
[Equation]
By construction, we have Pr ( A t = a t | ¯ S t =
¯ s t , ¯ Y [Equation] − 1 = ¯
y t − 1 , ¯ A t − 1 = ¯ a t − 1 ) > 0 .
Under SRA, the left-hand-side (LHS) of equation 14 equals [Equation] ∗ [Equation]
¯ a
[Equation]
−
[Equation] y [Equation] Notice that the conditioning event is the same as { S ∗ t (¯ a t − 1 )
= s t , Y ∗ t − 1 (¯ a t − 1 ) =
y t − 1 , ¯ S t − 1 = ¯ s t − 1 , ¯ A t − 1 = ¯ a t − 1 , ¯ Y t − 2 = ¯
y t − 2 } .
Under SRA, equation 15 equals
E { Q ( a (cid:48) ; a t , S ∗ t (¯ a t
[Equation]
[Equation] Y ∗
[Equation]
a
[Equation]
S
[Equation]
| S
[Equation]
By recuisvely applying SRA, we can show the left-hand-side of equation 14 equals
E [ Q ( a (cid:48) ; a t , S ∗ t (¯ a t − 1 )) − Y ∗ t (¯ a t ) − γQ ( a (cid:48) ; a (cid:48) , S ∗ t +1 (¯ a t ))
|{ S ∗ j (¯ a j − 1 ) =
s j } 1 ≤ j ≤ t , { Y ∗ j (¯ a j ) =
y j } 1 ≤ j ≤ t − 1 ] .
This is equal to zero by MA, CMIA and Lemma 2.
The proof is hence completed.
F. P ROOF OF T HEOREMF.2. P ROOF UNDER D
We begin by providing an outline of the proof.
The proof is divided into three steps.
In the ﬁrst step, we show for any T 1 ≤ t ≤ T k , the estimator (cid:98) β ( t ) satisﬁes the following linear representation,
[Equation]
where Σ ( t ) =
E (cid:98) Σ ( t ) and ε j,a = Y j + γQ ( a ; a, S j +1 )
− Q ( a ; A j , S j ) for a = 0 , 1 .
Based on this representation, in the second step, we show the asymptotic normality of (cid:98) τ ( t ) .
Speciﬁcally, we show
[Equation]
In the last step, we prove Theorem 3.
Part 1: By deﬁnition, we have
[Equation]
where the last equality is due to the deﬁnition of Σ j .
Let
[Equation]
It follows that
[Equation]
and hence
[Equation]
(cid:123)(cid:122) (cid:125)
ζ 2 ( t ) − (cid:98) Σ − 1 ( t )   t t − 1 (cid:88)
j = (cid:18)
ξ j r j,ξ j r j,(cid:19)  (cid:124) (cid:123)(cid:122) (cid:125) ζ 3 ( t ) .
We ﬁrst consider ζ 3 ( t ) .
It can be upper bounded by (cid:107) (cid:98) Σ − 1 ( t ) (cid:107)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 t t − 1 (cid:88)
{+j = (cid:18) ξ j ε j,ξ j ε j,(cid:19) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) = (cid:107) (cid:98) Σ − 1 ( t ) (cid:107) 2 max a ∈{ 0 , 1 } sup (cid:107) a (cid:107) 
 = (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) a (cid:62)   1 t t − 1 (cid:88)+}
j = ξ j r a,j  {+ (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:107) (cid:98) Σ − 1 ( t ) (cid:107) 2 max a ∈{ 0 , 1 } sup (cid:107) a (cid:107) 
 = (cid:118)(cid:117)(cid:117)(cid:116)t t − 1 (cid:88) j = ( a (cid:62) ξ j ) 2 r 2 a,j ≤ max a+}
[Table] where the second follows from Cauchy-Schwarz inequality.
Under Condition C2(i), we have for any j ≤ t ≤ T k , max j | r a,j | =
o ( t − 1 / 2 ) .
Suppose for now, we have shown [Equation] (cid:98) Σ − 1 ( t ) (cid:107)
= O p (1) and λ max
[Equation]
(cid:88)
[Equation]
It follows that
[Equation]
[Equation]
To bound ζ 2 ( t ) , notice that for any a ∈ { 0 , 1 } ,
[Equation]
Similar to Lemma 1, we can show for any ϕ ( · ) that is a function of ¯ A t , ¯ S t , ¯ Y t − 1 that [Equation] A
[Equation]
S t [Equation] This implies that E ξ (cid:62) j 1 ξ j 2 ε j 1 ,a ε j 2 ,a = 0 for any j 1 (cid:54) =
j 2 .
It follows that
[Equation]
Since all immediate rewards are uniformly bounded, so is the Q function.
As a result, | ε j,a | ’s are uniformly bounded.
Suppose for now, we have shown
[Equation]
It follows that E (cid:107) t − 1 (cid:80) t − 1 j =
ξ j ε j,a (cid:107) 22 = O ( q ) and hence
[Equation]
Suppose
[Equation]
It follows that
[Equation]
This together with equation 18 yields equation 16.
It remains to show equation 17, equation 20 and equation 21 hold.
We summarize these results in
Lemma 3.
Lemma 3 Under the given conditions, we have equation 17, equation 20 and equation 21 hold.
Part 2: By deﬁnition, we have
[Equation]
The asymptotic variance of √ t { (cid:98) τ ( t ) − τ 0 } is given by σ 2 ( t ) =
U (cid:62) Σ − 1 ( t ) Ω ( t ) { Σ − 1 ( t ) } (cid:62) U .
We begin by providing a lower bound for σ 2 ( t ) .
Notice that
[Equation]
Under C1(iii), we have lim inf q (cid:107) U (cid:107) 22 > 0 .
In addition, notice that Σ − 1 ( t ) { Σ − 1 ( t ) } − 1 is positive semi-deﬁnite.
It follows that λ min [ Σ − 1 ( t ) { Σ − 1 ( t ) } − 1 ] = 1 /λ max [ Σ ( t ) { Σ ( t ) } ] .
Using similar arguments in showing (cid:107) Σ (0) ∗ 2 , 2 (0) (cid:107)
= O (1) in the proof of Lemma 3, we can show sup t ≥ 1 (cid:107) Σ ( t ) (cid:107)
= O (1) and hence sup t ≥ 1 λ max [ Σ ( t ) { Σ ( t ) } ]
= O (1) .
This further yields
[Equation]
Suppose Ω ( t ) satisﬁes
[Equation]
It follows that σ 2 ( t ) is bounded away from zero, for sufﬁciently large t .
Under Condition C2(i), we have U (cid:62) β ∗ − τ
= o ( T − 1 / 2 k ) =
o ( t − 1 / 2 ) .
It follows that
[Equation]
Moreover, it follows from equation 22, equation 23 and equation 24 that σ ( t ) /
(cid:107) U (cid:107) 2 is uniformly bounded away from zero, for sufﬁciently large t .
Combining this together with equation 16 yields √
[Equation]
where the remainder term satisﬁes (cid:107) R t (cid:107)
= o p ( t − 1 / 2 ) .
It follows that the second term on the righthand-side (RHS) of the above expression is bounded from above by √ t (cid:107) R t
[Equation]
and hence
[Equation]
Similar to the proof of Lemma 1, we can show for any j ≥ 0 , a ∈ { 0 , 1 } ,
[Equation]
By the deﬁnition of ζ 1 ( t ) , √ t U (cid:62) ζ 1 ( t ) /σ ( t ) forms a martingle with respect to the ﬁltration σ ( { S j , A j , Y j } j<t ) , i.e. the σ -algebra generated by { S j , A j , Y j } j<t .
By the martingale central limit theorem, we can show √ t U (cid:62) ζ 1 ( t ) /σ
( t ) d → N (0 , 1) (see Lemma 4 for details).
To complete the proof of Part 2, we need to show equation 24 holds and that (cid:98) σ ( t ) /σ ( t ) P → 1 .
The assertion (cid:98) σ ( t ) /σ ( t ) P → 1 can be similarly proven using arguments from Step 3 of the proof of Theorem 1, Shi et al.
(2020a).
We show the asymptotic normality of √ t U (cid:62) ζ 1 ( t ) /σ
( t ) and that equation 24 holds in the following lemma.
Lemma 4 Under the given conditions, we have equation 24 holds and that √ t U (cid:62) ζ 1 ( t ) /σ ( t ) d →
N (0 , 1) .
Part 3: Results in Part 2 yield that √ T k { (cid:98) τ ( T k ) − τ 0 } /σ
( T k ) d → N (0 , 1) for each 1 ≤ k ≤ K .
In addition, for any K -dimensional vector a = ( a 1 , · · · , a K ) (cid:62) , it follows from equation 25 that
[Equation]
The leading term on the RHS can be rewritten as a weighted sum of { ξ j ε j, 0 , ξ j ε j, 1 } 0 ≤ j<t .
Similar to the proof in Part 2, we can show it forms a martingale with respect to the ﬁltration σ ( { S j , A j , Y j } j<t ) .
We now derive its asymptotic normality for any a , using the martingale central limit theorem for triangular arrays.
By Corollary 2 of McLeish (1974), we need to verify the following two conditions:
[Equation]
to some constant in probability.
Since K is ﬁxed, to verify (a), it sufﬁces to show
[Equation]
In Lemma 3, we have shown (cid:107) Σ − 1 ( t ) (cid:107) =
O (1) .
In Part 1 and Part 2 of the proof, we have shown | ε j,a | ’s are uniformly bounded and that σ ( t ) / (cid:107) U (cid:107) 2 is bounded away from zero.
Therefore, it sufﬁces to show T − 1 / [-
-]2max
[Equation]
j [Equation] j (cid:107) 2 P → 0 .
Under Condition C2(ii), we have sup s (cid:107) Ψ( s ) (cid:107)
= O ( q 1 / 2 ) and hence max 0 ≤ j<t (cid:107) ξ j (cid:107) 2 =
O ( q 1 / 2 ) .
The assertion thus follows by noting that T 1 = c 1 T and q = o ( T ) .
Using similar arguments in Step 3 of the proof of Shi et al.
(2020a), we can show
[Equation]
[Equation]
as t → ∞ .
This together with the facts (cid:107) Σ − 1 ( t ) (cid:107) =
O (1) and σ ( t ) / (cid:107) U (cid:107) 2 is bounded away from zero implies that
[Equation]
≤
[Equation]
P → 0 , where a ∧ b = min( a, b ) .
It follows that
[Equation]
o [Equation] In the proofs of Lemmas 3 and 4, we show that (cid:107) Σ − 1 ( t ) − ( Σ (0) ∗ ) − 1 (cid:107) 2 =
O ( t − 1 / 2 ) and (cid:107) Ω ( t ) − Ω (0) ∗ (cid:107) 2 =
O ( t − 1 / 2 ) for some matrices Σ (0) ∗ and Ω (0) ∗ that are invariant to t .
Deﬁnitions of these two matrices can be found in Sections F.5 and F.6.
Under C2(ii) and the condition that q = o ( √ T/ log T ) , we can show (cid:107) U (cid:107)
= O ( q 1 / 2 ) and hence σ 2 ( t ) P → ( σ (0) ∗ ) 2 where ( σ (0) ∗ )
2 =
U
[Equation] [Equation] [Equation]
Σ [Equation] [Equation] U .
Similar to equation 27, we have
[Equation]
→
[Equation]
[Table]
[Equation]
where c k ’s are deﬁned in Section 4.4.
This together with equation 27 yields that
[Equation]
Conditions (a) and (b) are thus veriﬁed.
By Lemma 4, we can show
[Equation]
for any ( a 1 , · · · , a K ) .
This yields the joint asymptotic normality of our test statistics.
By equation 28, its covariance matrix is given by Ξ 0 whose ( k 1 , k 2 ) -th entry is equal to ( c k 1 c k 2 ) − 1 / 2 c k 1 ∧ c k 2 .
Using similar arguments in proving equation 27, equation 28 and Stepof the proof in Theorem 1, Shi et al.
(2020a), we can show (cid:98) Ξ is a consistent estimator for Ξ 0 .
This completes the proof of Theorem 3 under D1.
F.2. P ROOF UNDER D
The proof is very similar to that under D1.
Suppose we can show equation 17, equation 20, equation 21 and equation 24 hold.
Then similar to the proof under D1, we have
[Equation]
The following lemma shows these assertions hold under D2 as well.
Lemma 5 Under the given conditions, we have equation 17, equation 20, equation 21 and equation 24 hold.
It follows that for any K -dimensional vector a =
( a 1 , · · · , a K ) (cid:62) ,
[Equation]
In the proof of Lemma 5, we show (cid:107){ Σ ( t ) } − 1 (cid:107) 2 =
O (1) , (cid:107) Σ ( t ) − Σ ∗ (cid:107)
= O ( t − 1 / 2 ) for some time-invariant matrix Σ ∗ that satisﬁes (cid:107) ( Σ ∗ ) − 1 (cid:107) 2 =
O (1) .
It follows that
[Equation]
(cid:107)
[Equation]
(cid:107){ Σ ( [Equation] [Equation] ( Σ ( t[-) − Ω ∗ (cid:107) 
-]
[Equation]
Σ
[Equation]
( Σ
[Equation]
2 ≤ (cid:107){ Σ ( t ) } − 1 (cid:107) 2 (cid:107) Σ ( t ) − Σ ∗ (cid:107) 2 (cid:107) ( Σ ∗ ) − 1 (cid:107) 2 =
O ( t − 1 / 2 ) .
{+Similarly, we can show (cid:107) Ω ( t ) − Ω ∗ (cid:107) 
+}
= O ( t − 1 / 2 ) for some matrix Ω ∗ .
In addition, using similar arguments in the proof of Lemma 5, we can show equation 26 holds under
D2 as well.
Now, the joint asymptotic normality of our test statistics follow using arguments from
Part 3 of the proof under D1.
Similarly, we can show (cid:98) Ξ is consistent.
This completes the proof under D2.
F.2. P ROOF UNDER D
The proof under D1 indicates that equation 17, equation 20, equation 21 and equation 24 hold with t =
T 1 .
It follows that
[Equation]
The rest of the proof is divided into two parts.
In the ﬁrst part, we show for k = 2 , · · · , K ,
[Equation]
for some ζ ∗ 1 ( T k ) and σ ∗ ( T k ) deﬁned below.
In the second part, we show the assertion in Theoremholds under D3.
Part 1:
For any 1 ≤ k ≤ K , consider the matrices
[Equation]
We show in Lemma 6 below that for k = 2 , · · · , K ,
[Equation]
and
[Equation]
where Σ ( k ) =
T − 1 k (cid:80) ki
=1 ( T i − T i − 1 )
Σ
( i ) .
Lemma 6 Under the given conditions, we have equation 31 and equation 32 hold.
Notice that ( T i − T
i − 1 ) /T
k
→ ( c i − c
i − 1 ) /c
k and (cid:107){ c − 1 k (cid:80) ki
=1 ( c i − c
i − 1 ) Σ ( i ) } − 1 (cid:107) 2 =
O p (1) where c 0 = 0 .
It follows from equation 31 that (cid:107) c − 1 k (cid:80) ki
=1 ( c i − c
i − [Equation] ( (cid:98) Σ ( i ) − Σ
( i ) )
(cid:107)
=
o p ( q − 1 / 2 ) and hence (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
T k k (cid:88) i
=
( T i − T i − 1 )( (cid:98) Σ ( i )
− Σ
i ) )
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
o p ( q − 1 / 2 ) , ∀ k =
[Equation] · · · , K.
Similar to the proof under D1, we can show (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:40)
T k k (cid:88) i
=
( T i − T i − 1 ) (cid:98) Σ ( i ) (cid:41) −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
=
O p (1) and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:40) 1 T k k (cid:88) i
=
( T i − T i − 1 ) (cid:98) Σ ( i ) (cid:41) −− { Σ ( k ) } − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
o p ( q − 1 / 2 ) , for k = 2 , · · · , K .
Thus, equation 21 and the ﬁrst assertion in equation 17 hold with t =
T 2 , T 3 , · · · , T K under D3.
In addition, similar to Lemma 6, we can show λ max
[Equation]
k −
[Equation]
(33) and (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)
T k − T k − T k − 1 (cid:88) j = T k −(cid:0) ξ j ξ (cid:62) j − E [ ξ j ξ (cid:62) j |{ ( S t , A t , Y t ) } 0 ≤ t<T k − 1 ] (cid:1) (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) =
o p ( q − 1 / 2 ) , for k = 2 , · · · , K .
This yields (cid:107) ( T k − T k − 1 ) − 1 (cid:80) T k − 1 j = T
k
− 1 ξ j ξ (cid:62) j (cid:107) 2 =
O p (1) .
As a result, the second assertion in equation 17 holds with t =
T 2 , T 3 , · · · , T K .
Moreover, using similar arguments in showing t − 1 (cid:80) t − 1 j =0 ( ξ (cid:62) j ε j, 0 , ξ (cid:62) j ε j, 1 ) (cid:62) =
O p ( t − 1 / 2 √ q ) under D1, we have by equation 33 that ( T k − T k − 1 ) − 1 (cid:80) T k − 1 j = T k − 1 ( ξ (cid:62) j ε j, 0 , ξ (cid:62) j ε j, 1 ) (cid:62) =
O p { ( T k −
T k − 1 ) − 1 / 2 √ q } , for k = 1 , · · · , K .
Under the given conditions on { T k } k , we obtain t − 1 (cid:80) t − 1 j =0 ( ξ (cid:62) j ε j, 0 , ξ (cid:62) j ε j, 1 ) (cid:62) =
O p ( t − 1 / 2 √ q ) for t =
T 2 , T 3 , · · · , T K .
Based on these results, using similar arguments in Part 1 of the proof under D1, we can show
[Equation]
[Equation]
where
[Equation]
For 1 ≤ k ≤ K , deﬁne
[Equation]
] , and Ω ( k )
=
T − 1 k (cid:80) ki
=1 ( T i − T i − 1 )
Σ
( i ) .
For any 2 ≤ k ≤ K , we have λ min ( Ω ( k ) ) ≥ λ min ( T − 1 k T 1 Ω (1) ) .
Since T − 1 k T 1 → c − 1 k c 1 > 0 and λ min ( Ω (1) )
=
λ min ( Ω ( T 1 )) is bounded away from zero, λ min ( Ω ( k ) ) is bounded away from zero for k
= 2 , · · · , K as well.
Deﬁne { σ
[Equation]
T k
[Equation]
U
[Equation]
Σ [Equation] U . It can be shown that σ ∗ ( T k ) / (cid:107) U (cid:107) 2 is bounded away from zero, for k = 2 , · · · , K .
Using similar arguments in Part 2 of the proof under D1, we can show equation 30 holds.
This completes the proof for Part 1.
Part 2: Let σ ∗ ( T 1 )
=
σ ( T 1 ) .
By equation 29 and equation 30, we have for any K -dimensional vector
a =
( a 1 , · · · , a K ) (cid:62) that
[Equation]
In the following, we show the leading term on the RHS of equation 35 is asymptotically normal.
Similar to the proof under D1, it sufﬁces to verify the following conditions:
[Equation]
to some constant in probability.
Condition (a) can be proven in a similar manner as in Part 3 of the proof under D1.
Notice that for k = 2 , · · · , K , Σ ( k ) , Ω ( k ) and σ ∗ ( T k ) are random variables and depend on the observed data history.
In the proof of Lemma 6, we show (cid:107) ( Σ ( k ) ) − 1 − ( Σ ∗∗ ) − 1 (cid:107)
= O p ( T − 1 / 2 ) for some deterministic matrix Σ ∗ and all k ∈ { 2 , · · · , K } .
Similarly, we can show (cid:107) Ω ( k ) − Ω ∗∗ (cid:107) 2 =
O p ( T − 1 / 2 ) and (cid:107){ σ ∗ ( T k ) } 2 − ( σ ∗∗ ) 2 (cid:107) 2 =
O p ( T − 1 / 2 ) for some Σ ∗ , σ ∗∗ and all k ∈ { 2 , · · · , K } .
Moreover, using similar arguments in the proof of Lemma 6, we can show (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
This further implies that
[Equation]
Based on these results, using similar arguments in Part 3 of the proof of Lemma 3, we obtain (b).
The joint asymptotic normality of √ T 1 { (cid:98) τ ( T 1 ) − τ 0 } /σ ∗ ( T 1 ) , · · · , √ T 1 { (cid:98) τ ( T 1 ) − τ 0 } /σ ∗ ( T K ) thus follows.
Consistency of (cid:98) Ξ can be similarly proven.
We omit the details for brevity.
F. P ROOF OF T HEOREM
As discussed in Section 4.3, ( Z ∗ 1 , Z ∗ 2 , · · · , Z ∗ K ) (cid:62) is jointly normal with mean zero and covariance matrix (cid:98) Ξ , conditional on the observed data.
By Theorem 1, we have (cid:98) Ξ P → Ξ 0 where Ξ 0 is the asymptotic covariance matrix of ( Z 1 , Z 2 , · · · , Z K ) (cid:62) .
Let α ∗ ( t )
=
α ( tT ) for any 0 ≤ t ≤ 1 , we have α ( T k ) → α ∗ ( c k ) for any 1 ≤ k ≤ K .
Notice that { (cid:98) b k } 1 ≤ k ≤ K is a continuous function of (cid:98) Ξ and { α ( T k ) } 1 ≤ k ≤ K , it follows that (cid:98) b k P → b k, 0 for 1 ≤ k ≤ K , where { b k, 0 } 1 ≤ k ≤ K are recursively deﬁned as follows:
[Equation]
where ( Z 1 , 0 , Z 2 , 0 , · · · , Z K, 0 ) (cid:62) is asymptotically normal with mean zero and covariance matrix Ξ 0 .
Theorem 3 implies that ( Z 1 − √ T 1 τ 0 / (cid:98) σ ( T 1 ) , Z 2 − √ T 2 τ 0 / (cid:98) σ ( T [Equation] [Equation]
[Equation]
σ
[Equation] [Equation]
( Z 1 , 0 , Z 2 , 0 , · · · , Z K, 0 ) (cid:62) .
It follows that
[Equation]
The proof is hence completed by noting that α ( T k ) → α ∗ ( c k ) .
When τ 0 = 0 , we have E Z k
= o (1) .
The rejection probability thus converges to the nominal level.
F. P ROOF OF T HEOREM
Suppose τ 0 =
T − 1 / 2 h for some h > 0 .
Based on the proof of Theorem 3, we can show (cid:98) σ ( T k ) P → σ ∗ k for some σ ∗ k > 0 .
It follows from equation 36 that
[Equation]
The second assertion in Theorem 2 thus holds by noting that α ( T k ) → α ∗ ( c k ) .
Let h → ∞ , we obtain
[Equation]
The proof is hence completed.
F. P ROOF OF L EMMA
Under the given conditions in C1(i), C1(ii) and C2(ii), equation 20 and the second assertion in equation 17 can be proven using similar arguments in the proof of Lemma E.2 and E.3 of Shi et al.
(2020a).
We omit the proof for brevity.
It remains to show equation 21 and the ﬁrst assertion in equation 17 hold.
Recall that µ is the density function of the stationary distribution Π (see the remark below Condition C1).
In addition, µ is uniformly bounded away from 0 and ∞ under C1(i).
For a (cid:48) ∈ { 0 , 1 } , deﬁne the matrix Σ (0) ∗ ( a (cid:48) ) = (cid:90) [Equation] (cid:88)
[Equation]
Deﬁne
[Equation]
The matrix Σ (0) ∗ is the population limit of (cid:98) Σ ( t ) under D1.
To prove the ﬁrst assertion in equation 17, we ﬁrst show
[Equation]
By deﬁnition, this is equivalent to show
[Equation]
for a ∈ { 0 , 1 } .
The matrix Σ (0) ∗ (0) can be written as
[Equation]
where
[Equation]
It follows that
[Equation]
and hence
[Equation]
[Table]
[Equation]
[Equation]
Thus, to prove (cid:107){ Σ (0) ∗ (0) } − {+1 (cid:107) 
+}
= O (1) , it sufﬁces to show
[Equation]
We ﬁrst consider equation 38.
Using similar arguments in Part 1 of the proof of Lemma E.2, Shi et al.
(2020a), it sufﬁces to show
[Equation]
for some ¯ c 1 > 0 .
Under D1, b (0) is strictly positive.
Since µ is strictly positive, it sufﬁces to show
[Equation]
for some ¯ c 2 > 0 .
Notice that LHS of equation 41 is equal to
[Equation]
where λ ( S ) is the Lebesgue measure of S .
Since S is compact, we have λ ( S ) < + ∞ .
By CauchySchwarz inequality, LHS of equation 41 is greater than or equal to λ
[Equation]
[Equation]
This is directly implied by Condition C2(ii).
The proof of equation 38 is hence completed.
Similarly, we can prove equation 39.
In addition, notice that
[Equation]
Since the density function µ is uniformly bounded
[Equation]
we have
[Equation]
where O (1) denotes the universal constant.
By Cauchy-Schwarz inequality
[Equation]
we obtain
[Equation]
In view of C2(ii), we obtain equation 40.
To summarize, we have shown (cid:107){ Σ (0) ∗ (0) } − 1 (cid:107)=
O (1) .
Similarly, we can prove (cid:107){ Σ (0) ∗ (1) } − 1 (cid:107)
= O (1) .
Assertion equation 37 thus holds.
Similar to Lemma E.5 of Shi et al.
(2020a), we can show (cid:107) Σ ( t ) − Σ (0) ∗ (cid:107)
= O ( t − 1 / 2 ) .
Using similar arguments in Partof the proof of Lemma E.2, Shi et al.
(2020a), this yields (cid:107) Σ − 1 ( t ) − ( Σ (0) ∗ ) − 1 (cid:107) [-
-]2 =
o ( t − 1 / 2 ) and (cid:107) Σ − 1 ( t ) (cid:107)
= O (1) .
Under the given conditions, equation 21 and the ﬁrst assertion in equationnow follow from the arguments used in Part 2 and 3 of the proof of Lemma E.2, Shi et al. (2020a).
F. P ROOF OF L EMMA
The asymptotic normality of √ t { (cid:98) τ ( t ) − τ 0 } /σ ( t ) can be proven using similar arguments in Partof the proof of Theorem 3.
In the following, we focus on equation 24.
Deﬁne the matrix Ω
[Equation]
Similar to Lemma E.5 of Shi et al.
(2020a), we can show (cid:107) Ω (0) ∗ − Ω ( t ) (cid:107)
= O ( t − 1 / 2 ) .
Thus, it sufﬁces to show inf q λ min ( Ω (0) ∗ ) > 0 .
Under CA and SRA, we have
[Equation]
For any q dimensional vectors a 1 , a 2 that satisfy (cid:107) a 1 (cid:107)
+ (cid:107) a 2 (cid:107)
= 1 , it follows that
[Equation]
=
[Equation]
a
[Equation]
] + {
[Equation]
a
[Equation]
[Equation]
[Equation]
a )
[Equation]
] + 2 { a
[Equation]
1 ξ ( s, a ) }{ a (cid:62) 2 ξ ( s, a ) } E [ { {+ε ∗ (0 , a ) ε ∗ (1 , a ) }| S 
+}
= s ] ≥ { a (cid:62) 1 ξ ( s, a ) }
[Equation]
E [ {
[Equation]
{+(0 , a ) } 2 | S 
+}
= s ]
+ { a (cid:62) 2 ξ ( s, a ) } 2 E
[ { ε ∗ (1 , a {+) } 2 | S 
+}
{+= s ] − 2 ρ ε | a (cid:62) 1 ξ ( s, a ) || a (cid:62) 2 ξ ( s, a ) | (cid:112) E [ { ε ∗ (0 , a ) } 2 | S 
 = s ] E [ { ε ∗ (1 , a ) } 2 | S 
+}
= s ] =
(1 − ρ ε ) { a (cid:62) 1 ξ ( s, a ) } 2 E [ { {+ε ∗ (0 , a ) } 2 | S 
+}
{+= s ] + (1 − ρ ε ) { a (cid:62) 2 ξ ( s, a ) } 2 E [ { ε ∗ (1 , a ) } 2 | S 
 = s ]+}
+ ρ ε (cid:12)(cid:12)(cid:12) | a (cid:62) 1 ξ ( s, a ) | (cid:112) E |{ ε ∗ (0 , a ) } 2 | S
= s ]
+ | a (cid:62) 2 ξ ( s, a ) | (cid:112) E [ { ε ∗ (1 , a ) }
| S
= s ] (cid:12)(cid:12)(cid:12)≥ (1 − ρ ε ) { a (cid:62) 1 ξ ( s, a ) } 2 E
[ { ε ∗ (0 , a ) } 2 | S
= s ] + (1 − ρ ε ) { a (cid:62) 2 ξ ( s, a ) } 2 E
[ { ε ∗ (1 , a ) } 2 | S
= s ] , where ρ ε = sup
q sup a ∈{ 0 , 1 } ,s ∈ S ρ ε ( a, s ) .
Under C3, we have ρ ε < 1 and inf q inf a (cid:48) ,a,s E [ { ε ∗ ( a (cid:48) , a ) } 2 | S
= s ] > 0 .
It follows that
[Equation]
a
[Equation]
, a
[Equation]
for some constant ¯ c 3 > 0 .
Therefore,
[Equation]
The strict positivity of µ ( · ) and the condition that b (0) ( · ) is uniformly bounded away from 0 andyields
[Equation]
for some constant ¯ c 4 > 0 .
With some calculation, we can show the RHS of equation 42 is equal to
[Equation]
By Condition C2(ii), it is strictly positive.
This yields inf q λ min ( Ω (0) ∗ ) > 0 .
Thus, we obtain equation 24.
F. P ROOF OF L EMMA
We begin by proving
[Equation]
[Equation] under D2.
For any matrices M 1 and M 2 , denote by diag [ M 1 , M 2 ] the block diagonal matrix (cid:20) M
M(cid:21) .
By MA and Condition C1(ii), the two Markov chains { S 2 t − 1 } t ≥ 1 , { S 2 t } t ≥ 0 are geometrically ergodic.
Let µ 1 and µ 2 denote the density function of their stationary distributions, respectively.
Under
C1(i), we can similarly show that they are uniformly bounded away from 0 and ∞ .
Deﬁne Σ
[Equation]
The matrix ( Σ ∗ 1 + Σ ∗ 2 ) / 2 corresponds to the population limit of Σ ( t ) .
Similar to Lemma E. of Shi et al.
(2020a), we can show (cid:107) Σ ∗ 1 − (2 t ) − 1 (cid:80) tj [-=
-]=0 E Σ 2 j +1 (cid:107)
= o ( t − 1 / 2 ) and (cid:107) Σ ∗ 2 − (2 t ) − 1 (cid:80) tj =0 E Σ 2 j (cid:107)
= o ( t − 1 / 2 ) .
This further yields
[Equation]
Similar to the proof of Lemma 3, in order to show equation 43, it sufﬁces to show (cid:107)
[Equation]
Notice that Σ ∗ 1 + Σ ∗ 2 = diag [ Σ ∗ (0) , Σ ∗ (1)] where Σ [Equation] s
[Equation]
The matrix Σ ∗ (0) can be further decomposed into
[Equation]
where Σ
[Equation]
Similar to the proof of Lemma 3, we can show (cid:107){ Σ ∗ 1 , 1 (0) } − 1 (cid:107)
= O (1) , (cid:107){ Σ ∗ 2 , 2 (0) } − 1 (cid:107) 2 =
O (1) and (cid:107) Σ ∗ 2 , 1 (0) (cid:107)
= O (1) .
It follows that (cid:107){ Σ ∗ (0) } − 1 (cid:107)
=
O (1) .
Similarly, we can show (cid:107){ Σ ∗ (1) } − 1 (cid:107) 2 =
O (1) .
This proves equation 44.
Thus, we obtain equation 43.
Using similar arguments in Part 2 of the proof of Lemma E.2, Shi et al.
(2020a), we can show (cid:107)[-t − 1 (cid:80) t − 1 j =0 Σ 2 j − Σ ∗ 2 (cid:107) 
-]
[-= O p ( t − 1 / 2 log t ) and (cid:107) t − 1 (cid:80) t − 1 j =0 Σ 2 j +1 − Σ ∗ 1 (cid:107) 
-][Equation]
=
[Equation] (cid:107) 2 = O p ( t [Equation]
1 / 2 log t ) and (cid:107) [Equation] This further implies (cid:107) (cid:98) Σ ( t ) − ( Σ
[Equation]
1 + Σ ∗ 2 ) / 2 (cid:107) = O p ( t − 1 / 2 log t ) and hence (cid:107) (cid:98) Σ ( t ) − Σ ( t ) (cid:107) 2 =
O p ( t − 1 / 2 log t ) .
Combining these results together with equation 43, we can show equation 21 and the ﬁrst assertion in equation 17 hold.
equation 20 and the second assertion in equation 17 hold can be proven in a similar manner.
Finally, using similar arguments in the proof of Lemma 4, we can show equation 24 holds.
We omit the details to save space.
F. P ROOF OF L EMMA
Under C1(iv), we have equation 6 holds.
Similar to equation 7, we can show Π ( k ) has a probability density function µ ( k ) given by
[Equation]
For a (cid:48) ∈ { 0 , 1 } , deﬁne Σ ( k ) ∗ ( [Equation] (cid:90)
[Equation]
(cid:88) [Equation] Condition on { ( S j , A j , Y j ) } 1 ≤ j<T k − 1 , the matrix Σ ( k ) ∗ ( a ) is deterministic.
Let Σ ( k ) =
diag [ Σ ( k ) ∗ (0) , Σ ( k ) ∗ (1)] .
Similar to the proof of Lemma 3, we can show (cid:107) Σ ( k ) ∗ − Σ ( k ) (cid:107)
= o (1) , conditional on { ( S j , A j , Y j ) } 1 ≤ j<T k − 1 , with probability tending to 1 .
This implies for any sufﬁciently small (cid:15) > 0 ,
[Equation]
The above conditional probability is bounded between 0 and 1 .
Using bounded convergence theorem, we have
[Equation]
and hence (cid:107) Σ ( k ) ∗ − Σ ( k ) (cid:107) 2 = o p (1) .
Notice that sup s | b ( k ) ( s ) − b ∗ ( s ) | P → 0 and (cid:107) Π ( k ) − Π ∗ (cid:107) TV P → 0 .
Deﬁne µ ∗ ( s ) =
(cid:88) a ∈
[Equation]
(cid:90) s ∈ S [ a { 1 − b ∗ ( s ) } + (1 − a ) b ∗ ( s )] p ( s (cid:48) ; a, s )Π ∗ ( ds ) . It follows that
[Equation]
ds
[Equation]
) | , and hence sup s | µ ( k ) ( s ) − µ ∗ ( s ) | P → 0 .
With some calculations, we can show (cid:107) Σ ( k ) ∗ ( a ) − Σ ∗ ( a ) (cid:107) 2 P → 0 where
[Equation]
Let Σ ∗ =
diag [ Σ ∗ (0) , Σ ∗ (1)] , we obtain (cid:107) Σ ( k ) ∗ − Σ ∗ (cid:107) 2 =
o p (1) .
Combining this together with equation 46, we obtain (cid:107) Σ ( k ) − Σ ∗ (cid:107)
= o p (1) .
The proof of Lemma 3 yields (cid:107) Σ (1) − Σ (0) ∗ (cid:107)
= o (1) .
Thus, we have for any 2 ≤ k ≤ K that
[Equation]
Similar to the proof of Lemma 3, we can show µ ( k ) ’s are uniformly bounded away from 0 and ∞ .
It follows that µ ∗ is uniformly bounded away from 0 and ∞ .
Using similar arguments in Lemma 3, we can show (cid:107){ T − 1 k T 1 Σ (0) [Equation]
+ T − 1 k ( T k − T 1 ) Σ ∗ } − 1 (cid:107)
= O (1) .
Using similar arguments in Partof the proof of Lemma E.2, Shi et al.
(2020a), we have by equation 47 that (cid:107) ( Σ ( k ) ) − 1 (cid:107)
= O (1) , with probability tending to 1 .
equation 32 is thus proven.
Assertion equation 31 now follows using similar arguments in Part 2 and Part 3 of the proof of
Lemma E.2, Shi et al. (2020a).
G A DDITIONAL FIGURES
We present some additional ﬁgures to report the simulation results in this section.
Figure 4 depicts the empirical rejection probabilities of the modiﬁed version of the O’Brien & Fleming sequential test developed by Kharitonov et al.
It can been seen that such a test has no power at all.
In addition, we remark that Kharitonov et al.
(2015)’s test requires equal sample size T 1 =
T k − T k −for k
= 2 , · · · , K and is not directly applicable to our setting with unequal sample size.
To apply such a test, we modify the decision time and set ( T 1 , T 2 , T 3 , T 4 , T 5 ) =
Figure 5 depicts the empirical rejection probabilities of our test and two-sample t-test with the error spending function given by α 2 .
Figure 6 reports the empirical rejection probabilities of our test with different combinations of the number of basis and the error spending function.
[Figure]