A N
O NLINE
S EQUENTIAL
T EST
FOR
Q UALITATIVE
T REATMENT E FFECTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Tech companies (e.g., Google or Facebook) often use randomized online experiments and/or A/B testing primarily based on the average treatment effects to compare their new product with an old one.
However, it is also critically important to detect qualitative treatment effects such that the new one may signiﬁcantly outperform the existing one only under some speciﬁc circumstances.
The aim of this paper is to develop a powerful testing procedure to efﬁciently detect such qualitative treatment effects.
We propose a scalable online updating algorithm to implement our test procedure.
It has three novelties including adaptive randomization, sequential monitoring, and online updating with guaranteed type-I error control.
We also thoroughly examine the theoretical properties of our testing procedure including the limiting distribution of test statistics and the justiﬁcation of an efﬁcient bootstrap method.
Extensive empirical studies are conducted to examine the ﬁnite sample performance of our test procedure.
I NTRODUCTION
Tech companies use randomized online experiments, or A/B testing to compare their new product with a well-established one.
Most works in the literature focus on the average treatment effects (ATE) between the new and existing products (see Kharitonov et al., 2015; Johari et al., 2015; 2017;
Yang et al., 2017; Ju et al., 2019, and the references therein).
In addition to ATE, sometimes we are interested in locating the subgroup (if exists) that the new product performs signiﬁcantly better than the existing one, as early as possible.
Consider a ride-hailing company (e.g., Uber).
Suppose some passengers are in the recession state (at a high risk of stopping using the companys app) and the company comes up with certain strategy to intervene the recession process.
We would like to if there are some subgroups that are sensitive to the strategy and pin-point these subgroups if exists.
It motivates us to consider the null hypothesis that the treatment effect is nonpositive for all passenger.
Such a null hypothesis is closely related to the notion of qualitative treatment effects in medical studies (QTE, Gail & Simon, 1985;
Roth & Simon, 2018; Shi et al., 2020a), and conditional moment inequalities in economics (see for example, Andrews & Shi, 2013; 2014; Chernozhukov et al., 2013;
Armstrong & Chan, 2016; Chang et al., 2015; Hsu, 2017).
However, these tests are computed ofﬂine and might not be suitable to implement in online settings.
Moreover, it is assumed in those papers that observations are independent.
In online experiment, one may wish to adaptively allocate the treatment based on the observed data stream in order to maximize the cumulative reward or to detect the alternative more efﬁciently.
The independence assumption is thus violated.
In addition, an online experiment is desired to be terminated as early as possible in order to save time and budget.
Sequential testing for qualitative treatment effects has been less explored.
In the literature, there is a line of research on estimation and inference of the heterogeneous treatment effects (HTE) (Athey & Imbens, 2016; Taddy et al., 2016; Wager & Athey, 2018; Yu et al., 2020).
In particular, Yu et al.
(2020) proposed an online test for HTE.
We remark that HTE and QTE are related yet fundamentally different hypotheses.
There are cases where HTE exists whereas QTE does not.
See Figure 1 for an illustration.
Consequently, applying their test will fail in our setting.
The contributions of this paper are summarized as follows.
First, we propose a new testing procedure for treatment comparison based on the notion of QTE.
When the null hypothesis is not rejected, the new product is no better than the control for any realization of covariates, and thus it is not useful at all.
Otherwise, the company could implement different products according to the auxiliary
[Figure]
covariates observed, to maximize the average reward obtained.
We remark that there are plenty cases where the treatment effects are always nonpositive (see Section 5 of Chang et al., 2015; Shi et al., 2020a).
A by-product of our test is that it yields a decision rule to implement personalization when the null is rejected (see Section 3.1 for details).
Although we primarily focus on QTE in this paper, our procedure can be easily extended to testing ATE as well (see Appendix D for details).
Second, we propose a scalable online updating algorithm to implement our test.
To allow for sequential monitoring, our procedure leverages idea from the α spending function approach (Lan &
DeMets, 1983) originally designed for sequential analysis in a clinical trial (see Jennison & Turnbull, 1999, for an overview).
Classical sequential tests focus on ATE.
The test statistic at each interim stage is asymptotically normal and the stopping boundary can be recursively updated via numerical integration.
However, the limiting distribution of the proposed test statistic does not have a tractable analytical form, making the numerical integration method difﬁcult to apply.
To resolve this issue, we propose a scalable bootstrap-assisted procedure to determine the stopping boundary.
Third, we adopt a theoretical framework that allows the maximum number of interim analyses K to diverge as the number of observations increases, since tech companies might analyze the results every few minutes (or hours) to determine whether to stop the experiment or continue collecting more data.
It is ultimately different from classical sequential analysis where K is ﬁxed.
Moreover, the derivation of the asymptotic property of the proposed test is further complicated due to the adaptive randomization procedure, which makes observations dependent of each other.
Despite these technical challenges, we establish a nonasymptotic upper bound on the type-I error rate by explicitly characterizing the conditions needed on randomization procedure, K and the number of samples observed at the initial decision point to ensure the validity of our test.
B ACKGROUND AND PROBLEM FORMULATION
We propose a potential outcome framework (Rubin, 2005) to formulate our problem.
Suppose that we have two products including the control and the treatment.
The observed data at time point t consists of a sequence of triples { ( X i , A
i , Y i ) } N ( t ) i =1 , where N ( · ) is a counting process that is independent of the data stream { ( X i ,
A
i , Y i ) } + ∞ i
=1 , A i is a binary random variable indicating the product executed for the i -th experiment,
X i ∈
R p denotes the associated covariates, and Y i stands for the associated reward (the larger the better by convention).
We allow A i to depend on X i and past observations { ( X j , A j , Y j ) }
j<i so that the randomization procedure can be adaptively changed.
In addition, deﬁne Y ∗ i (0) and Y ∗ i (1) to be the potential outcome that would have been observed if the corresponding product is executed for the i -th experiment.
Suppose that { ( X i , Y ∗ i (0) , Y ∗ i (1)) } + ∞ i = are independently and identically distributed copies of ( X, Y ∗ (0) , Y ∗ (1)) .
Let X be the support of
X and Q 0 ( x, a ) =
E { Y ∗ ( a ) | X = x } for a = 0 , 1 , we focus on testing the following hypotheses:
[Equation]
Notice that when there are no covariates, i.e., X = ∅ , the hypotheses are reduced to H 0 : τ 0 ≤versus H 1 : τ 0 > 0 , where τ 0 corresponds to ATE, i.e, τ 0 =
E { Y ∗ (1) − Y ∗ (0) } .
In general, we require X to be a compact set.
We consider a large linear approximation space Q for the conditional mean function Q 0 .
Speciﬁcally, let Q = { Q ( x, a ; β 0 , β 1 ) =
ϕ (cid:62) ( x ) β a : β 0 , β 1 ∈ R q } be the approximation space, where ϕ ( x ) is a q -dimensional vector composed of basis functions on X .
The dimension q is allowed to diverge with the number of observations in order to alleviate the effects of model misspeciﬁcation.
The use of linear approximation space simpliﬁes the computation of our testing procedure.
When Q 0 is well approximated, it sufﬁces to test
[Equation]
For clarity, here we assume Q 0 ( x, a ) =
Q ( x, a ; β ∗ 0 , β ∗ 1 ) for some β ∗ 0 and β ∗ 1 .
In Appendix B, we allow the approximation error inf β 0 ,β 1 ∈ R p sup x ∈ X ,a ∈{ 0 , 1 } | Q 0 ( x, a ) − Q ( x, a ; β 0 , β 1 ) | to be nonzero.
Let F j denote the sub-dataset { ( X i , A
i , Y i ) } 1 ≤ i ≤ j for j ≥ 1 and F 0 = ∅ .
Throughout this paper, we assume that the following two assumptions hold.
(A1) Y i
=
A i Y ∗ i (1) + (1 − A i )
Y ∗ i (0) for ∀ i ≥ 1 .
(A2)
A i is independent of Y ∗ i (0) , Y ∗ i (1) , { ( X k , Y ∗ k (0) , Y ∗ k (1)) }
k>i given X i and F i − 1 , for any i .
Assumption (A1) is referred to be the stable unit treatment value assumption (Rubin, 1974) and Assumption (A2) is the sequential randomization assumption (Zhang et al., 2013) and is automatically satisﬁed in a randomized study where the treatments are independently generated of the observed data. (A2) essentially assumes there is no unmeasured confounders.
These assumptions guarantee that both regression coefﬁcients (deﬁned through potential outcomes) are estimable from the observed dataset as shown in the following lemma.
Lemma 1 Let I ( · ) denotes the indicator function.
Under (A1)-(A2), we have
E [ I ( A i = a ) { Y i − ϕ (cid:62) ( X i ) β ∗ a } ] = 0 , ∀ a ∈ { 0 , 1 } , i ≥ 1 .
O NLINE SEQUENTIAL TESTING FOR QTE 3. T EST STATISTICS AND THEIR LIMITING DISTRIBUTION
We ﬁrst present our test statistic for testing H 0 .
In view of Lemma 1, we estimate β a by using the ordinary least squares estimator
[Equation]
  at each time point t for a ∈ { 0 , 1 } , where (cid:98) Σ a ( t ) =
N − 1 ( t ) (cid:80) N ( t ) i =
1 I ( A i = a ) ϕ
( X i ) ϕ
(cid:62) ( X i ) .
A generalized inverse might be used even if (cid:98) Σ a ( t ) is not invertible.
Consider the following test statistic
S ( t ) =
sup x ∈ X ϕ (cid:62) ( x ) { (cid:98) β 1 ( t ) − (cid:98) β 0 ( t ) } .
Under H 0 , we expect S ( t ) to be small.
A large S ( t ) can be interpreted as the evidence against H 0 .
As such, we reject H 0 for large S ( t ) .
We remark that when
H 0 is rejected, we can apply the decision rule d ( x ) =
arg max a ∈{ 0 , 1 } ϕ (cid:62) ( x ) (cid:98) β a ( t ) for personalized recommendation.
To determine the rejection region, we next discuss the limiting distribution of S ( t ) .
Under H 0 ,
[Equation]
Both equalities hold when β ∗
= β ∗ 1 .
Suppose there exists some function π ∗ ( · , · ) deﬁned on { 0 , 1 } × X that satisﬁes E X | (cid:80) ni =1 n − 1 π i − 1 ( a, X ) − π ∗ ( a, X ) | P → 0 , ∀ a ∈ { 0 , 1 } as n → ∞ , where π n ( · , · ) =
Pr ( A n = a | X n = x, F n − 1 ) , and the expectation E X is taken with respect to
X .
This condition implies that the treatment assignment mechanism cannot be arbitrary (see the discussion below Theorem 1 for details).
Then we will show
[Equation]
where Σ a = E π ∗ ( a, X ) ϕ ( X ) ϕ (cid:62) ( X ) , Φ a = E π ∗ ( a, X ) σ 2 ( a, X ) ϕ
( X ) ϕ
(cid:62) ( X ) , and σ 2 ( a, x ) =
E [ { Y ∗ ( a ) − ϕ (cid:62) ( X ) β a } 2 | X = x ] , for any x ∈ X .
According to equation 3, the right-hand-side (RHS) of equation 2 is to converge in distribution to the maximum of some Gaussian random variables.
This observation forms the basis of our test.
We next discuss the sequential implementation of our test.
Assume that the interim analyses are conducted at time points t 1 , t 2 , . .
, t K ∈ [0 , . . .
, T ] such that 0 < t 1 < t 2 < · · · < t K = T .
We allow K to grow with the number of observations.
In the most extreme case, one may set t k = inf t { N ( t ) ≥ N ( t k − 1 )+1 } , ∀ k ≥ 2 .
That is, we make a decision regarding the null hypothesis upon the arrival of each observation.
In addition, we assume that t 1 is large so that there are enough number of samples N ( t 1 ) to guarantee the validity of the normal approximation for B ( t 1 ) .
We remark that in typical tech companies such as Amazon, Facebook, etc., massive data are collected even within a short time interval.
Large sample approximation is validated in these applications.
To guarantee our test controls the type-I error, we reject H 0 and terminate the experiment at t k if (cid:112)
N
[Equation]
t k ) S (
[Equation]
k ) ≥ z k for some k = 1 , . . .
, K with some suitably chosen z 1 , . . .
, z K > 0 that satisfy
Pr
[Equation]
for a given signiﬁcance level α > 0 under H 0 .
In view of equation 2, it sufﬁces to ﬁnd { z k } k that satisfy
[Equation]
where the stochastic process B ( · ) is deﬁned in equation 3.
To determine { z k } k , we need to derive the asymptotic distribution of the left-hand-side (LHS) of equation 4.
To this end, deﬁne a mean-zero Gaussian process G ( t ) with covariance function
[Equation]
In the following, we show that the LHS of equation 4 can be uniformly approximated by G ( · ) , for any { z k } k =1 ,...,K .
To establish our theoretical results, we need some regularity conditions on ϕ ( · ) .
To save space, we summarize these assumptions in (A3) and put them in Appendix B.
Theorem 1 Assume (A1)-(A3) hold.
For a = 0 , 1 , assume inf x ∈ X π ∗ ( a, x ) > 0 and | Y ∗ ( a ) | is bounded almost surely.
Assume there exists some 0 < α 0 ≤ 1 such that for any sequence { j n } n that satisﬁes j α 0 n / log α 0 j n (cid:29) q 2 , the following event occurs with probability at least 1 − O
( j − α 0 n ) ,
[Equation]
(5) where O (1) denotes some positive constant.
Assume N α 0 ( t 1 ) / log α 0 N ( t 1 )
(cid:29) q
and N ( t 1 )
(cid:29) log N ( T ) almost surely.
Then conditional on the counting process N ( · ) , there exists some constant c > 0 such that
[Equation]
Theorem 1 implies that the approximation error depends on the number of observations obtained up to the ﬁrst decision point N ( t 1 ) , the maximum number of interim analyses K , the total number of basis functions q
[Equation]
and α 0 , which characterizes the convergence rate of the treatment assignment mechanism (cid:80) n i =1 n − 1 π i − 1 .
Clearly, the error will decay to zero when the followings hold with probability tending to 1 ,
[Equation]
[Equation]
In Appendix C, we show that α 0 = 1 / 2 , when an (cid:15) -greedy strategy is used for randomization to balance the trade-off between exploration and exploitation.
In this case, 6 requires q to grow at a slower rate than N 1 / 6 ( t 1 ) .
This condition is automatically satisﬁed when q is bounded.
Condition 7 is satisﬁed when K grows polynomially fast with respect to N ( t 1 ) .
In addition to (cid:15) -greedy, other adaptive allocation procedures (e.g., upper conﬁdence bound or Thompson sampling) could be applied as well.
As discussed in the introduction, the derivation of Theorem 1 is nontrivial.
One way to obtain the magnitude of the approximation error is to apply the strong approximation theorem for multidimensional martingales (see Morrow & Philipp, 1982; Zhang, 2004).
However, the rate of approximation typically depends on the dimension and decays fast as the dimension increases.
To derive Theorem 1, we view { ϕ (cid:62) ( x ) B ( t K ) } x ∈ X ,k ∈{ 1 ,...,κ } as a high-dimensional martingale and adopt the Gaussian approximation techniques that have been recently developed by Belloni & Oliveira (2018).
In view of equation 2, an application of Theorem 1 yields the following result.
Theorem 2 Assume that the conditions of Theorem 1 hold, equation 6 and equation 7 hold with probability tending to 1 .
Then for any z 1 , . . .
, z k that satisfy
[Equation]
as N ( t 1 ) diverges to inﬁnity, we have under H 0 ,
[Equation]
The above equality holds when β ∗
= β ∗ 1 .
Theorem 2 suggests that the type-I error rate of the proposed test can be well controlled.
It remains to ﬁnd critical values { z k } 1 ≤ k ≤ K that satisfy equation 8.
In the next section, we propose a bootstrapassisted procedure to determine these critical values.
3. B OOTSTRAP STOPPING BOUNDARY
We ﬁrst outline a method based on the wild bootstrap (Wu, 1986) to approximate the limiting distribution of { S ( t k ) } k .
Then we discuss its limitation and present our proposal, a scalable bootstrap algorithm to determine the stopping boundary.
The idea is to generate bootstrap samples { (cid:98) β MB a ( t k ) } a,k that have asymptotically the same joint distribution as { (cid:98) β a ( t k ) − β ∗ a } a,k .
Then the joint distribution of { S ( t k ) } k can be well-approximated by the conditional distribution of { (cid:98) S MB ( t k ) }
k given the data, where (cid:98) S MB ( t ) =
sup x ∈ X ϕ (cid:62) ( x ) { (cid:98) β MB 1 ( t ) − (cid:98) β MB 0 ( t ) } for any t .
Speciﬁcally, let { ξ i } + ∞ i =1 be a sequence of i.i.d.
standard normal random variables independent of { ( X i , A
i , Y i ) } + ∞ i =1 .
For a ∈ { 0 , 1 } , deﬁne (cid:98) β MB
[Equation]
Both the asymptotic means of (cid:112) N ( t ) (cid:98) β MB a ( t ) and (cid:112) N ( t )( (cid:98)
β a ( t ) − β ∗ a ) are zero.
In addition, their covariance functions are asymptotically the same.
By design, { (cid:98) β MB a ( t k ) } a,k is multivariate normal.
Similar to equation 3, we can show { (cid:98) β a ( t k ) − β ∗ a } a,k is asymptotically multivariate normal.
Consequently, the limiting distributions of { (cid:98) β MB a ( t k ) } a,k and { (cid:98) β a ( t k ) − β ∗ a } a,k are asymptotically equivalent.
As such, the bootstrap approximation is valid.
However, calculating (cid:98) β MB a ( t k ) requires O ( N ( t k )) operations.
The time complexity of the resulting bootstrap algorithm is O ( BN ( t k ))
up to the k -th interim stage, where B is the total number of bootstrap samples.
This can be time consuming when { N ( t k ) − N ( t k − 1 ) } Kk =1 are large.
To facilitate the computation, we observe that in the calculation of (cid:98) β MB a , the random noise is generated upon the arrival of each observation.
This is unnecessary as we aim to approximate the distribution of (cid:98) β a ( · ) only at ﬁnitely many time points.
We next present our proposal.
Let { e i,a } i =1 ,...,K,a =0 , 1 be a sequence of i.i.d N (0 , I q ) random vectors independent of the observed data, where I q denotes the q × q identity matrix.
At the kth interim stage, we compute (cid:98) S MB ∗ ( t k ) =
sup x ∈ X ϕ
[Equation]
( x ) { (cid:98) β
MB ∗( t k ) − (cid:98)
β
MB ∗( t k ) } , where (cid:98) β MB ∗ a ( t k ) equals
[Equation]

[Equation]
e
[Equation]
For any k 1 and k 2 , the conditional covariance of (cid:112) N ( t k 1 ) { (cid:98) β MB
[Equation] [Equation]
t
[Equation]
−
[Equation]
β
[Equation]
( t k 1 ) } and (cid:112)
N (
[Equation] [Equation]
( t k 2 ) } equals
[Equation]
Under the given conditions in Theorem 1, it is to converge to
[Equation]
This means { (cid:112) N ( t k )
( (cid:98) β MB* 1 ( t k ) − (cid:98) β MB
* 0 ( t k )) } k and { G ( t k ) }
k have the same asymptotic distribution.
Consequently, { (cid:112) N ( t k )
(cid:98) S MB ∗ ( t k ) }
Kk =1 can be used to approximate the joint distribution of { sup x ∈ X ϕ (cid:62) ( x ) G ( t k ) }
Kk =1 .
To choose { z k } k that satisﬁes equation 8, we adopt the α -spending approach that allocates the total allowable type I error at each interim stage according to an error-spending function.
This guarantees our test controls the type-I error.
We begin by specifying an α spending function α ( t ) that is nonincreasing and satisﬁes α (0)
= 0 , α ( T )
=
α .
Popular choices of α ( · ) include
[Equation]
[Equation]
where Φ( · ) denotes the cumulative distribution function of a standard normal variable and Φ − 1 ( · ) is its quantile function.
Based on α ( · ) , we iteratively calculate (cid:98)
z k , k = 1 , . . .
, K as the solution of
Pr ∗ (cid:26)
[Equation]
and reject H 0 when (cid:112) N ( t k ) S ( t k ) > (cid:98)
z k holds for some k .
The validity of the bootstrap test is summarized in Theorems 3 and 4 below.
Theorem 3 Assume the conditions in Theorem 1 hold.
Assume q
=
O
( N α ∗ ( t 1 )) for some 0 < α ∗ < 1 / 3 , almost surely.
Then conditional on the counting process N ( · ) , we have
[Equation]
KN
[Equation]
t 1 ) }
[Equation]
for some constant c > 0 with probability at least 1 − O ( N − α 0 ( t 1 )) , where Pr ∗ ( · ) denotes the probability measure conditional on the data stream { X i , A i , Y i } + ∞ i =1 .
Theorem 4 Assume the conditions in Theorem 3 hold.
Then conditional on N ( · ) , the critical values { (cid:98) z k } k satisfy
[Equation]
for some constant c > 0 .
When the RHS of equation 11 is o p (1) , it follows from Theorems 2 and 4 that our test is valid.
The conditional distribution in equation 10 can be approximated by the empirical distribution of
Bootstrap samples.
Finally, we remark that our test can be online updated as batches of observations arrive at the end of each interim stage.
A pseudocode summarizing our procedure is given in Algorithm 1 in the appendix.
The spatial complexity of the proposed algorithm is O ( B ) , where B is the number of bootstrap samples.
The time complexity is O
( Bk + N ( t k ))
up to the k -th interim stage.
Suppose
N ( t j ) − N ( t j − 1 ) = n for any 1 ≤ j ≤ K , we have Bk + N ( t k ) =
( B + n ) k
(cid:28) Bnk = BN ( t k ) for large n and B .
Hence, our procedure is much faster compared to the standard wild bootstrap.
N UMERICAL STUDIES 4. S IMULATION STUDIES
In this section, we conduct Monte Carlo simulations to examine the ﬁnite sample properties of the proposed test.
We generated the potential outcomes as Y ∗ i ( a )
= 1 + ( X i 1 − X i 2 ) / 2 + aτ
( X i )
+ ε i , where ε i ’s are i.i.d N (0 , 0 .
The covariates X i =
( X i 1 , X i 2 , X i 3 ) (cid:62) were generated as follows.
We ﬁrst generated X ∗ i =
( X ∗ i 1 , X ∗ i 2 , X ∗ i 3 ) (cid:62) from a multivariate normal distribution with zero mean and covariance matrix equal to { 0 .
5 | i − j | } i,j .
Then we set X ij =
X ∗ ij I ( X ∗ ij | ≤ 2) + 2 sgn ( X ∗ ij )
I ( X ∗ ij | > 2) .
We consider two randomization designs.
In the ﬁrst design, the treatment assignment is nondynamic and completely random.
Speciﬁcally we set π i ( a, x ) = 0 .
5 , for any a, x and i .
In the second design, we use an (cid:15) -greedy strategy to generate the treatment with ε = 0 .
In addition, we set N ( T 1 ) = 2000 and N ( T j ) − N ( T j − 1 ) =
2 n for 2 ≤ j ≤ K and some n > 0 .
We consider two combinations of ( n, K ) , corresponding to ( n, K ) =
(200 , 5) and (20 , 50) .
We set the signiﬁcance level α = 0 .
and choose B = 10000 .
We set τ ( X i ) =
φ δ { ( X i 1 +
X i 2 ) / √ 2 } X 2 i 3 for some function φ δ parameterized by some δ ≥ 0 .
We consider two scenarios for φ δ .
Speciﬁcally, we set φ δ ( x ) = δx 2 / 3 in Scenario 1 and φ δ = δ cos( πx ) in Scenario 2.
For each setting, we further consider four cases by setting δ = 0 , 0 . 1 , 0 . 15 , 0 . 2 , 0 . 25 and 0 .
When δ = 0 ,
H 0 holds.
Otherwise, H 1 holds.
For all settings, we construct the basis function ϕ ( · ) using additive cubic splines.
For each univariate spline, we set the number of internal knots to be 4 .
These knots are equally spaced between [ − 2 , 2] .
We denote our test by BAT, short for bootstrap-assisted test.
We run our experiments on a single computer instance with 40 Intel(R)
Xeon(R)
2.20GHz CPUs.
It takes 1-2 seconds on average to compute each test.
In Table 1 (see Appendix G), we report the rejection probabilities and average stopping times (deﬁned as the average number of samples consumed when the experiment is terminated) of the proposed test aggregated over 400 simulations when α 1 ( · ) is chosen as the spending function.
In Figure 2, we plot the rejection probabilities of our tests and the average stopping times of the experiments.
It can be seen that the type-I error rates are close to the nominal level in all cases.
The power of our test increases as δ increases, demonstrating its consistency.
In addition, when δ > 0 , our experiments are stopped early in all cases.
To further evaluate our method, we compare it with a test based on the law of iterated logarithm (denoted by LIL).
LIL determines the decision boundary based on an always valid ﬁnite error bound (see Appendix F for details about the competing method).
It can be seen from Figure 2 that our method has much larger power than the law of iterated logarithm approach.
4. R EAL DATA ANALYSIS
In this section, we apply the proposed method to a Yahoo!
Today Module user click log dataset 1 , which contains 45,811,883 user visits to the Today Module, during the ﬁrst ten days in May 2009.
For the i th visit, the dataset contains an ID of the new article recommended to the user, a binary response variable Y
i indicating whether the user clicked the article or not, and a ﬁve dimensional feature vector summarizing information of the user.
Due to privacy concerns, feature deﬁnitions and article names were not included in the data.
Each feature vector sums up to 1.
Therefore, we took the ﬁrst three and the ﬁfth elements to form the covariates X i .
For illustration, we only consider a subset of data that contains visits on May 1st where the recommended article ID is eitheror 109520.
These two articles were being recommended most on that day.
This gives us a total of 405888 visits.
On the reduced dataset, deﬁne A i
= 1 if the recommended article is 109510 and
A i = 0 otherwise.
We ﬁrst conduct A/A experiments (which compare these two articles against themselves) to examine the validity of our test.
The A/A experiments are done when every 2000 more users are available, we randomly assign 1000 users to arm A, and the other 1000 users in arm
B.
We expect our test will not reject H 0 in A/A experiments, since the articles being recommended are the same.
Then, we conduct
A/B experiment to test the QTE of these two articles.
The test statistics and their corresponding critical values are plotted in Figure 3.
On average it takes several seconds to implement our test.
It can be seen that our test is able to be reject H 0 after obtaining the ﬁrst one-third of the observations, in the A/B experiment.
In the A/A experiments, we fail to reject H 0 , as expected.
D ISCUSSION
In this paper, we propose a new testing procedure for evaluating the performance of technology products in tech companies based on the notion of qualitative treatment effects.
Currently, we only focus on comparing two products.
It would be practically useful to develop a multiple testing procedure under settings with multiple treatment options.
These topics warrant further investigations.
A N OTATIONS
We introduce some general notations used in the appendix.
For any matrix Mat, we use (cid:107) Mat (cid:107) p to denote the matrix norm induced by the corresponding (cid:96) p norm of vectors, for 1 ≤ p ≤ + ∞ .
For two nonnegative sequences { s 1 ,n } n and { s 2 ,n } n , we will use the notation s 1 ,n (cid:22) s 2 ,n to represent that s 1 ,n ≤ ¯ cs 2 ,n for some universal constant ¯ c > 0 whose value is allowed to change from place to place.
When a matrix Mat is degenerate, Mat − 1 denotes the Moore-Penrose inverse of Mat.
For any vector ψ , we use ψ ( i ) to denote its i -th element.
In Algorithm 1, we use O p +1 to denote a ( p + 1) ×
( p + 1) zero matrix and 0 p +1 to denote a ( p + 1) -dimensional zero vector.
B M ORE ON THE BASIS FUNCTION B.
[Equation]
(A3).
Assume λ min
[ E ϕ ( X ) ϕ (cid:62) ( X )] (cid:16) 1 , λ max [ E ϕ ( X ) ϕ (cid:62) ( X )] (cid:16) 1 , sup x (cid:107) ϕ
( x ) (cid:107)
= O ( q 1 / 2 ) , lim inf q inf x ∈ X (cid:107) ϕ
( x ) (cid:107) 2 > 0 .
In addition, assume
[Equation]
When a tensor-product B-spline is used (see Section 6 of Chen & Christensen, 2015, for a brief overview of tenor-product B-splines), (A3) is automatically satisﬁed.
Speciﬁcally, λ min [ E ϕ ( X ) ϕ (cid:62) ( X )] (cid:16) 1 , λ max [ E ϕ ( X ) ϕ (cid:62) ( X )] (cid:16) 1 follow from Theorem 3.3 of (Burman &
Chen, 1989).
sup x (cid:107) ϕ
( x ) (cid:107)
= O ( q 1 / 2 ) follows by noting that the absolute value of each element in ϕ ( x ) is bounded by some universal constant.
lim inf q inf x ∈ X (cid:107) ϕ
( x ) (cid:107) 2 > 0 follows from the arguments used in the proof of Lemma E.4 of Shi et al.
(2020b).
The last condition in equationholds by noting that each function in the vector ϕ ( · ) is Lipschitz continuous when a tensor-product
B-spline is used.
B. O N THE APPROXIMATION ERROR
The proposed test remains valid as long as the approximation error satisﬁes
[Equation]
with probability tending to 1 .
In the following, we introduce some sufﬁcient conditions for equation 13.
Suppose the Q-function Q 0 ( · , a ) is p -smooth (see the deﬁnition of p -smoothness in Stone, 1982), for a ∈ [0 , 1] .
When a tensor-product B-spline or Wavelet basis is used for ϕ ( · ) , then there exist some β ∗ 0 and β ∗ 1 that satisfy [Equation]
[Equation]
See Section 2.2 of Huang (1998) for detailed discussions on the approximation power of these basis functions.
Condition equation 13 is thus automatically satisﬁed when q (cid:29) { N ( T ) } d/ (2 p ) , with probability tending to 1 .
C A DAPTIVE RANDOMIZATION
In practice, the company might want to allocate more trafﬁc to a better treatment based on the observed data stream.
The (cid:15) -greedy strategy is commonly used to balance the trade-off between exploration and exploitation.
For a given 0 < ε 0 < 1 , consider the following randomization procedure: for some integer N 0 > 0 and any j ≥ N 0 , a ∈ { 0 , 1 } , x ∈ X , we set
[Equation]
where
[Equation]
It is immediate to see that (cid:98) Σ a ( t )
= (cid:98) Σ a,n ( t ) and (cid:98) β a ( t ) =
(cid:98) β a,n ( t ) .
Deﬁne π
[Equation]
( x )( β 1 − β 0 ) ≤ 0 } for any a ∈ { 0 , 1 } and x ∈ X .
Lemma 2 Assume (A1)-(A3) hold.
Assume inf x ∈ X π ∗ ( a, x ) > 0 and | Y ∗ ( a ) | is bounded almost surely, for a ∈ { 0 , 1 } .
Assume Pr ( | ϕ (cid:62) ( X )( β 1 − β 0 ) | ≤ (cid:15) ) ≤ L 0 (cid:15) , for some constant L 0 > 0 and any (cid:15) > 0 .
Then for any { j n } n that satisﬁes √ j n / √ log j n (cid:29) q 2 , the following event occurs with probability at least 1 − O ( j − 1 n ) ,
[Equation]
Lemma 2 implies that Condition equation 5 in Theorem 1 automatically holds with α 0 = 1 / 2 , when the epsilon-greedy strategy is used.
When ϕ (cid:62) ( X )( β 1 − β 0 ) is a continuous random variable, the assumption Pr ( | ϕ (cid:62) ( X )( β 1 − β 0 ) | ≤ (cid:15) ) ≤ L 0 (cid:15) for any (cid:15) > 0 in Lemma 2 is satisﬁed if ϕ (cid:62) ( X )( β 1 − β 0 ) has a bounded probability density function.
When ϕ (cid:62) ( X )( β 1 − β 0 ) is discrete, this assumption is satisﬁed if inf x ∈ X | ϕ (cid:62) ( x )( β 1 − β 0 ) | > 0 .
D T ESTING THE AVERAGE TREATMENT EFFECTS D. T HE ALGORITHM
We focus on testing the following hypothesis,
H 0 : E Y ∗ i (1) ≤ E Y ∗ i (0) versus H 1 : E Y ∗ i (1)
>
E Y ∗ i (0) .
Under (A1) and (A2), it sufﬁces to test
H 0 : E Q ( X i , 1) ≤ E Q ( X i , 0) versus H 1 : E Q ( X i , 1) > E Q ( X i , 0) .
We similarly use basis approximations to model the Q-function.
Our proposal is summarized in the following algorithm.
We next conduct simulation studies to evaluate this algorithm.
Input: Number of bootstrap samples B , an α spending function α ( · ) .
Initialize: n = 0 , (cid:98) Σ
= (cid:98) Σ 1 = O p +1 , (cid:98) γ
= (cid:98) γ 1 =
p
+1 , (cid:98) β 0 ,b
= (cid:98) β 1 ,b =
p
+1 , ¯ ϕ = 0 and a set
I = { 1 , . . .
, B } .
For k = 1 to K do
Initialize: m = 0 , (cid:98) φ = 0 and (cid:98) Φ
= (cid:98) Φ 1 =
O p +1 .
For i = N ( t k − 1 ) + 1 to N ( t k ) do n = n + 1 , m = m
+ 1 and ¯ ϕ = n − 1 ( n − 1)
¯ ϕ + n − 1 ϕ
( X i ) ; (cid:98) Σ
a = (1 − n − 1 ) (cid:98) Σ a + n − 1 ϕ
( X i ) ϕ
(cid:62) ( X i )
I
( A i = a ) , a = 0 , 1 ; (cid:98) γ
a = (1 − n − 1 ) (cid:98) γ a + n − 1 ϕ
( X i )
Y
i
I
( A i = a ) , a = 0 , 1 ;
Compute (cid:98) β a = (cid:98) Σ − 1 a (cid:98) γ a for a ∈ { 0 , 1 } and S = ¯ ϕ (cid:62) ( (cid:98) β 1 − (cid:98) β 0 ) ;
For i = N ( t k − 1 ) + 1 to N ( t k ) do (cid:98) φ
= (cid:98) φ + [ { ϕ ( X i ) − ¯ ϕ } (cid:62) ( (cid:98) β 1 − (cid:98) β 0 )] 2 .
(cid:98) Φ a = (cid:98)
Φ a + (cid:98) Σ − 1 a ϕ
( X i ) ϕ
(cid:62) ( X i ) {
Y i − ϕ (cid:62) ( X i ) (cid:98) β a } 2 (cid:98) Σ − 1 a
I ( A i = a ) , a = 0 , 1 ;
For b = 1 , . . .
B do
Generate two independent N (0 , I p +1 )
Gaussian vectors e 0 , e 1 , N (0 , 1) random variable e 2 ; (cid:98) β a,b =
(1 − mn − 1 ) (cid:98) β a,b + n − 1 (cid:98) Φ 1 / 2 a e a + n − 1 (cid:98) φ 1 / 2 e 2 , a = 0 , 1 ;
Compute (cid:98) S b = ¯ ϕ (cid:62) ( (cid:98) β 1 ,b − (cid:98) β 0 ,b ) ;
Set z to be the upper { α ( t ) − |I| c /B |} /
(1 − |I c | /B )
-th percentile of { (cid:98) S b } b ∈I ;
Update I as I ← { b ∈ I : (cid:98) S b ≤ z } .
If S > z :
Reject H 0 and terminate the experiment;
D. N UMERICAL STUDIES
In this section, we compare our procedure with the always valid test for testing ATE (Johari et al., 2015).
We generate the potential outcomes with the same model, except that ε i ’s are i.i.d N (0 , 1) .
However, we set N ( T 1 ) = 1000 and N ( T j ) − N ( T j − 1 ) =
2 n for 2 ≤ j ≤ K and some n > 0 .
We consider two combinations of ( n, K ) , corresponding to ( n, K ) =
(100 , 5) and (10 , 50) .
For all settings, we use a linear function to approximate Q .
In Table 2 (see Appendix G) and Figure 4, we show the rejection probabilities and average stopping times of the proposed test aggregated over 400 simulations, when α 1 ( · ) is chosen as the spending function.
It can be seen that our method behaves better than the always valid test when the effect size is small, and comparable when the effect size is large.
The always valid test fails in the adaptive randomization settings, as the type-I error rates are around 50% under the null hypothesis.
E P ROOFS
Note that we require X to be a compact set.
To simplify the proof, we assume X = [0 , 1] d .
E. P ROOF OF L EMMA
Set F 0 = ∅ .
We state the following lemma before proving Lemma 1.
Lemma
For any j ≥ 1 , ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ F j − 1 .
For any a ∈ { 0 , 1 } , i ≥ 1 , notice that
[Equation]
where the ﬁrst equation is due to Assumption (A1) and
E X i , F i − 1 denotes the conditional expectation given F i − 1 and X i .
By Assumption (A2), we have
[Equation]
The second term on the RHS equals zero due to Lemma 3 and our model assumption
E { Y ∗ i ( a ) | X i } =
ϕ (cid:62) ( X i ) β a .
The proof is hence completed.
E. P ROOF OF T HEOREM
Let n ( · ) be the realization of the counting process N ( · ) .
We will show the assertion in Theoremholds for any such realizations that satisfy n ( t 1 ) < n ( t 2 ) < · · · < n ( t K ) .
The case where some of the n ( t k ) ’s are the same can be similarly discussed.
For any j ≥ 1 , deﬁne σ ( F j ) to be the σ -algebra generated by F j .
For a ∈ { 0 , 1 } , deﬁne (cid:98) Σ a,j = 1 j j (cid:88) i =
I ( A i = a ) ϕ
( X i ) ϕ
(cid:62) ( X i ) and (cid:98) β a,j = (cid:98) Σ − 1 a,j (cid:32)j j (cid:88) i =
I ( A i = a ) ϕ
( X i )
Y i (cid:33) .
It is immediate to see that (cid:98) Σ a ( t )
= (cid:98) Σ a,n ( t ) and (cid:98) β a ( t ) =
(cid:98) β a,n ( t ) .
Deﬁne δ n = qn − α 0 log α 0 n .
We state the following lemmas before proving Theorem 1.
Lemma 4 There exists some constant 0 < (cid:15) 0 < 1 such that λ min [ E ϕ ( X ) ϕ (cid:62) ( X )] ≥ (cid:15) 0 , λ max [ E ϕ ( X ) ϕ (cid:62) ( X )] ≤ (cid:15) − 1 0 , sup x (cid:107) ϕ
( x ) (cid:107) 2 ≤ sup x (cid:107) ϕ ( x ) (cid:107) 1 ≤ (cid:15) − 1 0 √ q , min a ∈{ 0 , 1 } λ min [Σ a ] ≥ (cid:15) 0 , max a ∈{ 0 , 1 } (cid:107) β a (cid:107) 2 ≤ (cid:15) − 1 0 , max a ∈{ 0 , 1 } | Y ∗ ( a ) | ≤ (cid:15) − 1 0 and sup x max a ∈{ 0 , 1 }
| ϕ (cid:62) ( x ) β a | ≤ (cid:15) − 1 0 .
Lemma 5 Assume the conditions in Theorem 1 hold.
Then for any sequence { j n } n that satisﬁes j α 0 n / log α 0 ( j n ) (cid:29) q 2 , we have with probability at least 1 − O ( j − α 0 n ) that for any a ∈ { 0 , 1 } and any k ≥ j n ,
[Equation]
Lemma 6 Assume the conditions in Theorem 1 hold.
The for any sequence { j n } n that satisﬁes j n / log( j n ) (cid:29) q , we have with probability at least 1 − O ( j − 1 n ) that for any a ∈ { 0 , 1 } and any k ≥ j n ,
[Equation]
=
[Equation]
For a ∈ { 0 , 1 } , (cid:98)
[Equation]
a
[Equation]
a = (cid:98)
[Equation]
1 a (cid:34)k
[Equation]
(cid:88) i =
I (
[Equation]
a
[Equation]
X
[Equation]
Y
[Equation]
− ϕ
[Equation]
X
[Equation] [Equation]
} (cid:35) , and hence (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:98)
[Equation]
k
[Equation]
a (cid:34)k
[Equation]
(cid:88) i =
I ( A
[Equation]
= a ) ϕ
( X
[Equation]
) { Y i − ϕ (cid:62) ( X i ) β a } (cid:35)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
with probability at least 1 − O ( j − α 0 n ) , by Lemma 5 and Lemma 6.
Deﬁne
[Equation]
It follows that
[Equation]
17) with probability at least 1 − O ( n − α
( t 1 ))
[Equation]
and hence (cid:107) sup
[Equation]
log n
[Equation]
(cid:112)
[Equation]
∀ k
[Equation]
1 , with probability at least 1 − O ( n − α
( t 1 )) , for some constant ¯ c > 0 , by equation 41.
Under the given conditions on q and n ( t 1 ) , we have
[Equation]
and hence
[Equation]
Thus, for any given z 1 , z 2 , . . .
, z K , we obtain
[Equation]
where
[Equation]
For any i ≥ 1 , 1 ≤ k ≤ K , deﬁne a q -dimensional vector
[Equation]
i,k =
[Equation]
or equivalently,
[Equation]
[Equation] [Equation]
≤ n (
[Equation]
k )) , by Condition (A1).
Let ξ i
=
( ξ (cid:62) i, 1 , ξ (cid:62) i, 2 , · · · , ξ (cid:62) i,K ) (cid:62) and M j = (cid:80) ji =1 ξ i .
The sequence {M i } i ≥forms a multivariate martingale with respect to the ﬁltration { σ ( F i ) :
i ≥ 1 } , since
[Equation]
by (A2).
Let n ( t 0 ) = 0 .
For any i such that n ( t k − 1 ) < i ≤ n ( t k ) for some 1 ≤ k ≤ K , we have (cid:107) ξ i (cid:107)
[Equation]
where the second inequality is due to Lemma
[Equation]
Therefore,
[Equation]
It follows that
[Equation]
Deﬁne a sequence of independent Gaussian vectors { η i } i ≥ 1 that satisfy η i ∼ N (0 , E ( ξ i ξ (cid:62) i |F i − 1 )) for any
i ≥ 1 .
Then the distribution of η i is the same as
[Equation]
where Z is a p -dimensional mean-zero Gaussian vector with covariance matrix
[Equation]
where the second equality follows from (A2) and Lemma 3, the third equality is due to the deﬁnition of π i − 1 and the last equality follows from Lemma 3.
Similar to equation 19, we can show that
[Equation]
Using similar arguments in equation 20, we can show that for any 1 ≤ k 1 ≤ k 2 ≤ K , n ( t K )
[Equation]
=
[Equation]
Let
V ( k 1 , k
[Equation]
=
[Equation]
Consider an arbitrary sequence of R p
+1 vectors { b k } 1 ≤ k ≤ K .
Under the given conditions, we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) b (cid:62)
[Equation]
  n
[Equation]
K )
[Equation]
i =
E
[Equation]
ξ
[Equation]
F
[Equation]

 b
[Equation]
2 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:22) n ( t k 1 ) (cid:88) a ∈{
[Equation]
1 } (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n
( t k 1 )
[Table]
=
[Table]
Deﬁne a matrix V as
V =  
V (1 , 1)
V (1 , 2) . . .
V (
[Equation]
K )
V (2 , 1)
V (2 , 2) . . .
V (2 , K ) ...
V ( K, 1) V ( K, 2) . . .
V ( K, K )   .
It follows that (cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13) n ( t K ) (cid:88) i =
E (
[Equation]
V (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) .
Using similar arguments in proving equation 14, we can show the RHS of the above equation is upper bounded by
[Equation]
and hence by (cid:15) − 2 0 qδ n ( t 1 ) , with probability at least 1 − O ( n − α 0 ( t 1 )) .
Therefore, we have
[Equation]
(23) with probability at least 1 − O ( n − α 0 ( t 1 )) , where I Kp ×
Kp denotes a Kp ×
Kp identity matrix.
[Table]
is bounded between 0 and 1 .
For any a ∈ { 0 , 1 } and any z > 0 , we have
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]

[Equation]
Under the given conditions, we have
E
[Equation]
Therefore, we obtain
[Equation]
or
[Equation]
(24) since n − α 0 ( t 1 ) (cid:28) δ n ( t 1 ) .
Combining equation 19 with equation 21, equation 23 and equation 24, an application of Theorem 2.1 in Belloni & Oliveira (2018) yields that
[Equation]
for any thrice differential function ψ ( · ) , and
[Equation]
where ∂ j g ( z ) denotes the partial derivative ∂g ( z ) /∂z ( j ) for any function g ( · ) and z ( j ) stands for the j -th element of z .
Let X k, 0 be an ε -net of X that satisﬁes the following: for any x ∈ X , there exists some x 0 ∈ Xsuch that (cid:107) x − x 0 (cid:107) 2 ≤ ε .
Set ε = √ d/n 4 ( t 1 ) .
Since X = [0 , 1] d , there exists some X 0 with
[Equation]
n
[Equation]
(26) where | X 0 | denotes the number of elements in X 0 .
Under Condition (A3), we have
[Equation]
It follows that
[Equation]
Using similar arguments in showing equation
[Equation]
we can show the following event occurs with probability at least 1 − O ( n − 1 ( t 1 )) ,
[Equation]
This together with equation 27 yields
[Equation]
with probability at least 1 − O ( n − 1 ( t 1 )) .
Under the given conditions
[Equation]
we have n ( t 1 ) (cid:29) max( q, log n ( t K )) .
It follows that there exists some constant ¯ c ∗ > 0 such that
[Equation] [Equation]
with probability at least 1 − O ( n − 1 ( t 1 )) .
Deﬁne
[Equation]
Combining equation 28 with equation 18 yields
[Equation]
Notice that M n ( t
[Equation]
By equation 26 and Lemma 4, there exist a set of vectors d 1 , d 2 , . . .
, d L ∈ R qK with L ≤ n 4 d ( t 1 ) K , max j (cid:107) d j (cid:107) 1 ≤ (cid:15) − 1 0 q 1 / 2 and a function k ( · ) that maps { 1 , . . .
, L } into { 1 , . . .
, K } such that
[Equation]
for any { ν k } Kk =1 .
For any η > 0 , m ∈ R qK , consider the function φ η, { ν k } k : R
qK → R , deﬁned as
[Equation]

[Equation]
It has the following property:
[Equation]
It follows that
[Equation]
where
[Equation]
The value of δ will be speciﬁed later.
In addition
[Equation]
with some calculations
[Equation]
we have
[Equation]
[Table]
[Equation]
Since max i (cid:107) d i (cid:107) 1 ≤ (cid:15) − 1 0 q 1 / 2 , we obtain that
[Equation]
By Lemma 5.1 of Chernozhukov et al.
(2016), for any δ > 0 , there exists some function g δ ( · ) : R →
R with (cid:107) g (cid:48) δ (cid:107) ∞
≤ δ − 1 , (cid:107) g (cid:48)(cid:48) δ (cid:107) ∞
≤ K 0 δ − 2 , (cid:107) g
cid:48)(cid:48)(cid:48) δ (cid:107) ∞ ≤ K 0 δ − 3 for some constant K 0 >
such that
[Equation]
It follows that
[Equation]
for any m ∈ R qK .
Combining this together with equation 30, equation 31 and equation
[Equation]
we obtain that
[Equation]
Consider the function g δ ◦ φ η, { ν k } k .
Apparently, we have
[Equation] [Equation]
By equation 33, we can show that
[Equation]
Set δ =
η
[Equation]
{ log
[Equation]
d log n ( t 1 ) }
[Equation]
we obtain
[Equation]
Combining equation 37 together with equation 25 and equation 36 yields
[Equation]
This together with equation 34 and equation 35 yields
[Equation]
Similar to equation 31-equation
[Equation]
we can show
[Equation]
where z
[Equation]
k, + = z ∗ k, + + η − 1 { log K + 4 d log n ( t 1 ) } + 3 δ and z ∗∗∗ k, − =
z ∗ k, − − η − 1 { log K + 4 d log n ( t 1 ) } − 3 δ, for each k .
Notice that for any { ν k } k , we have
[Equation]
This together with equation 38 and equation 39 yields
[Equation]
In view of equation
[Equation]
we have shown that
[Equation]
The covariance matrix Cov( G ( t k )) is given by (cid:80) a ∈{ 0 , 1 } Σ − 1 a Φ a Σ − 1 a and is nonsingular by Lemma 4.
In addition, we have (cid:107) ϕ
( x ) (cid:107) 2 ≥ ¯ c, ∀ x ∈ X 0 , by Condition A3.
Thus, there exists some constant c ∗ > 0 such that
[Equation]
By Theorem 1 of Chernozhukov et al.
[Equation]
we obtain that
[Equation]
Thus, we obtain
[Equation]
3 η 3 { log 3 K + log 3 n ( t 1 ) } + q 2 η 2 { log 2 K + log 2 n ( t 1 ) } δ n ( t 1 ) + n − α 0 ( t 1 ) + η − 1 { log K + log n ( t 1 ) } 3 / 2 + qδ n ( t 1 ) { log K + log n ( t 1 ) } 1 / 2 + (cid:112) qn − 1 ( t 1 ) log n ( t 1 ) { log K + log n ( t 1 ) } 1 / 2 .
Setting η = min( q − 3 / 4 n 1 / 8 ( t 1 ) log − 3 / 8 { Kn ( t 1 ) } , q − 1 n − α 0 / 3 ( t 1 ) log − α 0 / 3 − 1 / 6 { Kn ( t 1 ) } ) yields the desired results.
The proof is hence completed.
E. P ROOF OF L EMMA
The assertion trivially holds for j = 1 .
We prove it holds for any j ≥ 2 , by induction.
By (A2), we have ( X j , Y ∗ j (0) , Y ∗ j (1))
⊥⊥
A 1 | X 1 .
Since ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ ( X 1 , Y ∗ 1 (0) , Y ∗ 1 (1)) , this further implies ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ A 1 and hence ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ ( X 1 , A 1 , Y ∗ 1 (0) , Y ∗ 1 (1)) .
By (A1), Y 1 is completely determined by A 1 , Y ∗ 1 (0) and Y ∗ 1 (1) .
Therefore, we obtain ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ F 1 .
Suppose we have shown that ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ F k for some k < j − 1 .
To prove ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ F k +1 , it sufﬁces to show ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ ( X k +1 , A k +1 , Y k +1 ) .
By (A1), Y k +1 is determined by A k +1 , Y ∗ k +1 (0) and Y ∗ k +1 (1) .
Since ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ ( X k +1 , Y ∗ k +1 (0) , Y ∗ k +1 (1)) , it sufﬁces to show ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ A k
+1 .
This is implied by ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ A k +
| X k +1 , F k and that ( X j , Y ∗ j (0) , Y ∗ j (1)) ⊥⊥ X k +1 , F k .
The proof is hence completed.
E. P ROOF OF L EMMA
The assertions
[Equation]
(40) and
[Equation]
for some 0 < (cid:15) 0 < 1 are directly implied by the conditions that λ min [ E ϕ ( X ) ϕ (cid:62) ( X )] (cid:16) 1 , λ max [ E ϕ ( X ) ϕ (cid:62) ( X )] (cid:16) 1 , sup x (cid:107) ϕ
( x ) (cid:107) 1 ≤ (cid:15) − 1 0 √ q .
Since (cid:107) ϕ
( x ) (cid:107) 2 ≤ (cid:107) ϕ
( x ) (cid:107) 1 , we obtain sup x (cid:107) ϕ
( x ) (cid:107) 2 ≤ sup x (cid:107) ϕ ( x ) (cid:107) 1 ≤ (cid:15) − 1 0 √ q .
Under the condition inf a,x π ∗ ( a, x ) > 0 , we can similarly show that λ min [Σ a ] ≥ (cid:15) 0 for some (cid:15) 0 > 0 .
Since | Y ∗ (0) | and | Y ∗ (1) | are bounded, there exists some constant 0 < (cid:15) 0 < 1 that satisﬁes max a ∈{ 0 , 1 } | Y ∗ ( a ) | ≤ (cid:15) − 1 0 .
Notice that ϕ (cid:62) ( x ) β a = E { Y ∗ ( a ) | X = x } .
Boundedness of | Y ∗ ( a ) | implies that the conditional mean E { Y ∗ ( a ) | X } is a bounded random variable as well.
As a result, we obtain sup x ∈ X max a ∈{ 0 , 1 } | ϕ (cid:62) ( x ) β a | ≤ (cid:15) − 1 0 .
Notice that β a = Σ − 1 a E ϕ (cid:62) ( X ) Y ∗ ( a ) .
Since λ min [Σ a ] is bounded away from 0 , it sufﬁces to show (cid:107) E ϕ (cid:62) ( X ) Y ∗ ( a ) (cid:107)
= O (1) , or equivalently, sup
[Equation]
By Cauchy-Schwarz inequality
[Equation]
it sufﬁces to show
[Equation]
Since | Y ∗ ( a ) | =
O (1) almost surely
[Equation]
we have by the condition λ max [ E ϕ ( X ) ϕ (cid:62) ( X )]
= O (1) that sup
[Equation]
The proof is hence completed.
E. P ROOF OF L EMMAE.5. P ROOF OF EQUATION
Notice that
[Equation]
1 , X ) (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
By Lemma 4, we have
[Equation]
with probability at least 1 − O ( j − α 0 n ) .
Consider the ﬁrst term on the RHS of equation 42.
For any i ≥ 1 , deﬁne M i = ϕ
( X i ) ϕ
(cid:62) ( X i ) {
A i − π i − 1 (1 , X i ) } .
Notice that { M i } i ≥ 1 forms a martingale difference sequence with respect to the ﬁltration { σ ( F i − 1 ) :
i ≥ 2 } , since
[Equation] [Equation]
where E F i ,X i denotes the conditional expectation given X i and F i .
Here, the ﬁrst equality is due to that X i ⊥⊥ F i − 1 , implied by Lemma 3.
Under the given conditions on the basis function ϕ ( · ) , using similar arguments in proving Equation (C.15) of Shi et al. (2020b), we can show that the following event occurs with probability at least 1 − O ( j − 2 ) ,
[Equation]
Notice that (cid:80) k ≥ j k − 2 ≤ j − 2 + (cid:80) k>j { k ( k − 1) } − 1 =
j − 2 + j − 1 .
Thus, the following occurs with probability at least 1 − O
( j − 1 n ) ,
[Equation]
It follows that
[Equation]
with probability at least 1 − O ( j − α 0 ) .
Similarly, we can show
[Equation]
with probability at least 1 − O ( j − α 0 n ) .
The proof is hence completed.
E.5. P ROOF OF EQUATION
When j n satisﬁes j α 0 n / log α 0 ( j n ) (cid:29) q 2 , it follows from equation 14 and equation 40 that
[Equation]
with probability at least 1 − O ( j − α 0 n ) .
Combining equation 40 with equation 45 and equation 14, we obtain
[Equation]
with probability at least 1 − O ( j − α 0 n ) .
The proof is hence completed.
E. P ROOF OF L EMMA
For any l ∈ { 1 , . . .
, q }
and i ≥ 1 , deﬁne M i ( l ) =
ϕ ( l ) ( X i )
A i { Y i − ϕ (cid:62) ( X i ) β 1 } .
Here, ϕ ( l ) ( X i ) corresponds to the l -th element of ϕ ( X i ) .
Similar to equation 43, we can show { M i ( l ) } i ≥ 1 forms a martingale difference sequence with respect to the ﬁltration { σ ( F i − 1 ) :
i ≥ 1 } .
By equation 43, we have for any l ,
[Equation]
Notice that
[Equation]
where the ﬁrst equality is due to (A1), the ﬁrst inequality is due to that A is bounded between 0 and 1 , the second equality follows from Lemma 3, the second inequality follows from Lemma 4, and the last inequality is due to equation 46.
It follows that
[Equation]
Similarly, by (A1) and Lemma 4, we have
[Equation]
Similar to equation 44, we can show with probability at least 1 − O ( j − 1 )
that
[Equation]
Thus, for any sequence j n that satisﬁes j n / log( j n ) (cid:29) q , we have by equation 47 that
[Equation]
, for some constant ¯ c > 0 , with probability at least 1 − O ( j − 1 n ) .
It follows that
[Equation]
}   

[Equation] [Equation]
  
 ≥ 1 − Pr    
(cid:91) k ≥ j n {| k (cid:88) i =
M i ( l ) | > 2 (cid:112) ¯ ck log k }   (cid:92)   (cid:92) k ≥ j n { k (cid:88) i =
[ M 2 i ( l ) + { M 2 i ( l ) |F
i − 1 } ] ≤ ¯ ck }   
 − O ( j − 1 n ) .
By Bonferroni’s inequality and Theorem 2.1 of Bercu & Touati (2008), we have
Pr   (cid:92) k ≥ j n {| k (cid:88) i =
M i ( l ) | ≤ 2 (cid:112) ¯ ck log k } 
 ≥ 1 − O
( j − 1 n ) − (cid:88) k ≥ j n
Pr   {| k (cid:88) i =
M i ( l ) | > 2 (cid:112) ¯ ck log k } (cid:92)   (cid:92) k (cid:48) ≥ j n { k (cid:48) (cid:88) i =
[ M 2 i ( l ) + { M 2 i ( l ) |F
i − 1 } ] ≤ ¯ ck (cid:48) }  

 ≥ 1 − O ( j − 1 n ) − (cid:88) k ≥ j n
Pr (cid:32) {| k (cid:88) i =
M i ( l ) | > 2 (cid:112) ¯ ck log k } (cid:92) (cid:40) k (cid:88) i =
[ M 2 i ( l ) + { M 2 i ( l ) |F
i − 1 } ] ≤ ¯ ck (cid:41)(cid:33) ≥ 1 − O
( j − 1 n ) − 2 (cid:88)
[Equation]
The last term on the RHS of equation 49 is 1 − O ( j − 1 n ) .
To summarize, we have shown that the following event occurs with probability at least 1 − O ( j − 1 n ) , (cid:92)
[Equation]
By Bonferroni’s inequality, we have (cid:92)
[Equation]
with probability at least 1 − O ( j − 1 / 2 n ) .
Similarly, we can show
[Equation]
, for some constant c > 0 , with probability at least 1 − O ( j − 1 n ) .
The proof is hence completed.
E. P ROOF OF T HEOREM
We state the following lemmas before presenting the proof.
Lemma 7 Assume the conditions in Theorem 3 hold.
Then for any sequence { j n } n that satisﬁes j α
[Equation]
n / log
[Equation]
j n
[Equation]
q 2 , we have with probability at least 1 − O ( j − α 0 n ) that
[Equation]
Lemma 8 Assume the conditions in Theorem 3 hold.
Then for any sequence { j n } n that satisﬁes j α 0 n / log α 0 j n (cid:29) q 2 , we have with probability at least 1 − O ( j − α 0 n ) that
[Equation]
Similar to the proof of Theorem 1, we will show the assertion in Theorem 3 holds for any n ( · ) that correspond to the realizations of N ( · ) that satisfy n ( t 1 ) < n ( t 2 ) < · · · < n ( t K ) .
For any 1 ≤ k 1 ≤ k 2 ≤ K , deﬁne
[Equation]
=
[Equation] [Equation]
and
[Equation]

[Equation] [Equation]
[Equation]

[Equation]
We aim to bound the entrywise (cid:96) ∞ norm of (cid:98) V − V where V is deﬁned in equation 22.
It sufﬁces to bound max
[Equation]
max
[Equation]
V ( k 1 , k 2 ) − V ( k 1 , k 2 ) (cid:107) 2 .
For any k 1 , k 2 , we decompose (cid:98) V ( k 1 , k 2 ) − V ( k 1 , k 2 ) as (cid:98) V ( k 1 , k 2 ) − V ( k 1 , k 2 ) =
(cid:98) V ( k 1 , k 2 ) − (cid:98) V ∗ ( k 1 , k 2 )
+ (cid:98) V ∗ ( k 1 , k 2 ) − (cid:98) V ∗∗ ( k 1 , k 2 ) + (cid:98) V ∗∗ ( k 1 , k 2 ) − V ( k 1 , k 2 ) , where (cid:98) V ∗ ( k 1 , k 2 ) =
[Equation]
(cid:112)
n
[Equation] [Equation]
n ( t k 2 ) 1 (cid:88) a = k 1 (cid:88) j = n ( t j ) (cid:88) i = n ( t j − 1 )+ Σ − 1 a
I
( A i = a ) ϕ
( X i ) ϕ
(cid:62) ( X i ) {
Y i − ϕ (cid:62) ( X i ) (cid:98) β a ( t j ) } 2 Σ − 1 a , (cid:98) V ∗∗ ( k 1 , k 2 ) =
(cid:112) n ( t k 1 ) n ( t k 2 ) 1 (cid:88) a = n ( t k 1 ) (cid:88)
j = Σ − 1 a I ( A i = a ) ϕ ( X i ) ϕ (cid:62) ( X i )
{ Y i − ϕ (cid:62) ( X i ) β a } 2 Σ − 1 a .
By Lemma 4 and Lemma 8, we obtain that max
[Equation]
max
[Equation]
(cid:13)(cid:13) n ( t k 1 ) n ( t k 1 ) (cid:88)
j = Σ − 1 a I ( A i = a ) ϕ ( X i ) ϕ (cid:62) ( X i )
{ Y i − ϕ (cid:62) ( X i ) β a } 2 Σ − 1 a − Σ − 1 a Φ a Σ − 1 a (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)≤ max 1 ≤ k 1 ≤ K (cid:15)1 (cid:88) a = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n ( t k 1 ) n ( t k 1 ) (cid:88)
j =
I ( A i = a ) ϕ
( X i ) ϕ
(cid:62) ( X i ) {
Y i − ϕ (cid:62) ( X i ) β a } 2 − Φ a (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:22) qδ n ( t 1 ) + q 1 / 2 n − 1 / 2 ( t 1 ) (cid:112) log n ( t 1 ) , (50) with probability at least 1 − O ( n − α
( t 1 )) .
Notice that
[Equation]
It follows that
[Equation]
By Lemma 4, we obtain that
[Equation] [Equation]
max
[Equation] [Equation]
(cid:88)
j = n (
[Equation]
j ) (cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + max
[Equation]
n ( t
[Equation]
(cid:13)(cid:13) (cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k 1 (cid:88)
j = n ( t j ) (cid:88) i = n ( t j − 1 )
+
I ( A i = a ) ϕ
[Equation]
By Lemmas 4 and
[Equation]
we have with probability at least 1 − O ( n − 1 ( t 1 ))
that
[Equation]
Similar to Lemma
[Equation]
we can show there exists some constant c ∗ > 0 that
[Equation]
with probability at least 1 − O ( n − 1 ( t 1 )) .
By Lemma 4, we can show with probability at least 1 − O ( n − 1 ( t 1 ))
that
[Equation]
This together with equation 52 and equation 53 yields
[Equation]
with probability at least 1 − O ( n − 1 ( t 1 )) .
Moreover, using similar arguments in proving Equation (C.15) of Shi et al. (2020b), we can show that for any 1 ≤ k 1 ≤ K , the following event occurs with probability at least 1 − O ( n − 2 ( t k 1 )) ,
[Equation]
[Equation] [Equation]
Since (cid:80) Kk 1 =1 n − 2 ( t k 1 ) ≤ n − 1 ( t 1 ) , we obtain with probability at least 1 − O ( n − 1 ( t 1 ))
that
[Equation]
In addition, it follows from Lemma 7 that
[Equation]
This together with equation 54 yields that
[Equation]
with probability at least 1 − O ( n − 1 ( t 1 )) .
Under the given conditions, we have
[Equation] [Equation]
with probability at least 1 − O ( n − 1 ( t 1 )) .
Moreover
[Equation]
with some calculations
[Equation]
we can show that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
In view of Lemma 4 and Lemma 5, we have with probability at least 1 − O ( n − α
( t 1 ))
that
[Equation]
where O (1) denotes some positive constant.
Similar to equation 50 and equation 55, we can show with probability at least 1 − O ( n − α 0 ( t 1 ))
that
[Equation]
Similar to Lemma 4, we can show max a ∈{ 0 , 1 } (cid:107) Ψ a (cid:107)
= O (1) .
It follows that
[Equation]
with probability at least 1 − O ( n − α
( t 1 )) .
Combining this together with equation 50 and equation 55, we obtain with probability at least 1 − O ( n − α 0 ( t 1 ))
that
[Equation]
Consider the function g δ ◦ φ η, { ν k } k deﬁned in the proof of Theorem 1.
We ﬁx δ = η − 1 { log K + 4 d log n ( t 1 ) } .
Based on Lemma A2 in Belloni & Oliveira (2018), we have with probability at least 1 − O ( n − α
( t 1 ))
that
[Equation]
where E ∗ denotes the expectation conditional on the observed data
[Equation]
For a given set of thresholds { ν k } k , using similar arguments in proving equation 31, equation 32, equation 34 and equation
[Equation]
we can show with probability at least 1 − O ( n − α 0 ( t 1 ))
that
[Equation]
and
[Equation]
where O (1) denotes some positive constant
[Equation]
and
[Equation]
By Theorem 2 of Chernozhukov et al.
[Equation]
we obtain that
[Equation]
( t k ) − ν ∗ k, + (cid:19) ≤(cid:27) − Pr (cid:26) max k ∈{ 1 ,...,K } (cid:18) sup x ∈ X 0 ϕ (cid:62) ( x ) G ( t k ) − ν ∗ k, − (cid:19) ≤(cid:27) (cid:22)
η − 1 { log 3 / 2 n ( t 1 ) + log 3 / 2 K } + ¯ c ∗ n − 2 ( t 1 ) { log 1 / 2 n ( t 1 ) + log 1 / 2 K } .
It follows that
[Equation]
with probability at least 1 − O ( n − α
( t 1 )) .
Set
[Equation]
we obtain the desired result.
E. P ROOF OF L EMMA
Combining Lemma 6 with Lemma 4 yields that
[Equation]
[Equation]
with probability at least 1 − O ( j − 1 n ) .
Combining this together with equation 16 yields that
[Equation]
with probability at least 1 − O ( j − 1 n ) .
The proof is hence completed.
E
[Equation]
P ROOF OF L EMMA
Notice that
[Equation]
Similar to the proof of Lemma 5, we can show that the second term on the RHS of equation 56 is of the order
O ( qδ k + (cid:112) qk − 1 log k ) , for any a ∈ { 0 , 1 } and any k ≥ j n , with probability at least 1 − O
( j − α 0 n ) .
As for the ﬁrst term, notice that each element in the matrix
[Equation]
corresponds to a martingale with respect to the ﬁltration { σ ( F i − 1 ) :
i ≥ 1 } , under (A1) and (A2).
Using similar arguments in proving Equation (C.15) of Shi et al.
(2020b), we can show that
[Equation]
with probability at least 1 − O ( j − 1 n ) .
The proof is hence completed.
E. P ROOF OF L EMMA
We begin by providing an upper bound for max a ∈{ 0 , 1 }
[Equation]
β a
[Equation]
− β
[Equation]
With some calculations
[Equation]
we have
[Equation]
By Lemma 6, we obtain with probability at least 1 − O
( j − 1 n ) that
[Equation]
Similarly, we can show with probability at least 1 − O ( j − 1 n )
that
[Equation]
Similar to equation
[Equation]
we have
[Equation]
Using similar arguments in proving equation
[Equation]
we can show that
[Equation] [Equation]
∀ k ≥ j n , with probability at least 1 − O
( j − 1 n ) .
Similarly, we can show max a
[Equation] [Equation]
i
[Equation]
with probability at least 1 − O ( j − 1 n ) .
Without loss of generality, assume ε 0 ≤ 1 / 2 .
Notice that we have π i − 1 ( a, x ) ≥ ε 0 , for any a ∈ { 0 , 1 } , x ∈ X
and i ≥ N 0 .
This together with Lemma equation 4 implies that
[Equation]
Combining this together with equation 60 and equation 61 yields
[Equation]
for some constant L ∗ ≥ 1 , with probability at least 1 − O ( j − 1 n ) .
This together with equation 58 and equation 59 yields that
[Equation]
with probability at least 1 − O ( j − 1 n ) .
By Condition (A3), we have
[Equation]
for some constant ¯ L > 0 , with probability at least 1 − O ( j − 1 n ) .
For any z 1 , z 2 ∈ R , we have I ( z 1 > 0) (cid:54) =
I ( z 2 > 0) only when | z 1 − z 2 | ≥ | z 2 | .
Hence, under the event deﬁned in equation 62, the event I { ϕ (cid:62) ( X )( (cid:98) β 1 ,k −
[Equation]
k ) > 0 } (cid:54) =
I { ϕ (cid:62) ( X )( β 1 − β 0 ) > 0 } occurs only when |
[Equation]
for any k ≥ j n .
Under the given conditions, we have
[Equation]
Notice that when I { ϕ (cid:62) ( X )( (cid:98) β 1 ,k − (cid:98) β 0 ,k ) > 0 } =
I { ϕ (cid:62) ( X )( β 1 − β 0 ) > 0 } , we have π k ( a, X ) =
π ∗ ( a, X ) .
Thus, we obtain π k ( a, X ) =
π ∗ ( a, X ) if | ϕ (cid:62) ( X )( β 1 − β 0 ) | > ¯ Lqk − 1 / 2 (cid:112) log max( k, j n ) , for any k ≥ L ∗ √ q log j n .
Set k
=
L ∗ √ q log j n .
By equationand equation 63, we have with probability at least 1 − O ( j − 1 n ) that
[Equation]
The proof is hence completed.
F C OMPARISON OF THE BASELINE
Consider our test statistic S ( t ) .
Under H 0 , it can be bounded from above by
[Equation]
It sufﬁces to provide an upper bound for the above expression.
By Cauchy-Schwarz inequality, equation 64 can be upper bounded by
[Equation]
It sufﬁces to provide anytime upper bound for (cid:107)
[Equation]
Recall that
[Equation] [Equation]
The above expression is asymptotically equivalent to
N
[Equation]
By the law of iterated logarithm
[Equation]
the (cid:96) -th dimension of the above expression can be upper bounded by
[Equation]
where (cid:80) (cid:96) (cid:98) σ 2 (cid:96) can be consistently estimated by
N ( t )
[Equation]
As such
[Equation]
the ﬁnite error bound is given by
[Equation]
N
[Equation]
[Table]
G A DDITIONAL TABLES AND FIGURES
[Table] [Table]