A Balanced Design of Time Series Experiments
Anonymous Author(s)
AffiliationAddress email
Abstract
Time series experiments are a family of experimental designs on a time series.
Oneexperimental unit is sequentially exposed to some version of treatment, stays in theversion of treatment for a duration of time, and gets exposed to another version oftreatment.
While this type of experimental designs could handle population interference between units, it typically still needs to account for temporal interference,i e., a treatment at an earlier period persists in impacting the outcomes of the laterperiods.
Practitioners have widely recognized the applicability of the time seriesexperiments, yet prior work typically requires a long duration to gain enough power.
In this paper, we propose a novel randomized design that significantly increasesthe power of such experiment.
We prove the theoretical performance of the noveldesign, and verify its superior performance by conducting an extensive simulationstudy
Introduction
Time series experiments — where we expose one aggregated unit to random treatments, measure theresponses, and repeat the procedure for some time periods — have rapidly grown in popularity [5].
This growth has been partly propelled by marketplace companies, such as DoorDash [29], Lyft [7],and Uber [9], wishing to run experiments in the presence of population interference (the setting whereone unit’s treatment assignments impact another’s outcomes).
To overcome population interference,companies aggregate multiple units together across zip codes, cities, or even states to form a singlegrouped unit [6, 12, 16, 18].
The aggregation alleviates the population interference, as it ensures thateach unit within the aggregated unit receives the same treatment, but it does not remove the temporalinterference (where past treatment assignments impact current outcomes).
Unfortunately, such adrastic aggregation significantly reduces the sample size as we only collect one data point at eachperiod, leading to much lower statistical power for estimating causal effects.
Therefore, it is vitalthat the design of time series experiments has a high data-efficiency – not too many time periods arerequired to increase the power of estimating the causal effect of interests.
In this work, we present a new, more powerful, randomized design for time series experiments subjectto temporal interference.
Our proposed design incorporates a randomization mechanism inspired bythe classical completely randomized design to achieve the balancing between treatment and controlobservations.
We analyze the theoretical property of our balanced design, develop efficient inferentialmethods and evaluate the performance using simulation study.
Setups and notations2.1 The assignment path and the potential outcomes
Consider an experimenter running time series experiments over one large aggregated unit.
Let therebe a total of T ∈ N periods in the experiment.
As suggested by earlier works [6, 29], each period is typically selected to be the same length as the carryover effect, i.e., the length of time a treatmentpersists in impacting the outcome.
For on-demand service platforms, one period typically ranges fromabout 30 minutes to several hours [7, 29].
At each time period t ∈ [ T ] , the experimenter randomlyexposes the unit to either treatment W t = 1 or control W t = 0 .
The assignment path is the collectionof treatment assignments over all time periods, W 1: T ∈ { 0 , 1 } T .
We denote a realization of W 1:
Tby w 1: T .
Once the assignment path w 1: T is realized, the experimenter will observe some outcomes of interests.
Under the potential outcomes framework [25, 14, 17], we model the observed outcomes to be relatedto their respective potential outcomes.
We denote Y t ( w 1: T ) to be the outcome at time period t ∈
[ T ]under the assignment path w 1: T .
As a short-hand notation, denote Y =
{ Y t ( w 1: T ) } t, w 1: T to be thecollection of all potential outcomes.
We adopt a design-based perspective [23, 11, 19, 26, 17, 1] andtreat the potential outcomes as fixed quantities.
The potential outcome Y t ( w 1: T ) is very general in the sense that it depends on the complete assignment path.
If experimenter directly observes the outcome Y t ( w 1: T ) under one assignment path w 1: T ,it becomes impossible to observe the potential outcomes under other assignment paths.
In other words,all the remaining data are missing.
Since we do not assume any structural models for characterizingthe outcomes, it is impossible to achieve valid inference with such missing data.
Therefore, we needto introduce some assumptions that restrict the dependence of the potential outcomes.
2.2 Temporal interference
In this section we introduce the assumption about the temporal interference, which lays the foundations for our design of time series experiments.
We consider a form of temporal interference betweentime periods, which is also known as the carryover effects .
We assume that the potential outcome atany time period depends only on the treatment assignments at this time period
period, but not
Assumption 1 (Limited carryover effects) .
For any t ∈ [ T ] and any two assignment pathsw 1: T , w ′ 1: T ∈ { 0 , 1 } T , we have
[Equation]
w
[Equation]
= w
[Equation]
t
[Equation]
This assumption is both widely adopted in the literature [20, 27, 2, 5, 6] and viable in many applications.
We can relax this assumption by considering general lengths of carryover effects, i.e.,the potential outcomes at one time period could depend on the treatment assignments up to m > 1periods ago.
The structure of the main results remains the same.
In this paper, we focus on the caseof m = 1 to de-emphasize the importance of the length of carryover effects to our main results, as theexisting literature [6, 29] recommends selecting the length of one time period to be the same as thelength of the carryover effect.
In a ride-sharing platform, the effect of a surge pricing policy quicklyvanishes and for instance, the carryover effect lasts 30 - 60 minutes.
We would set the length of aperiod to be one hour so that the length of the carryover effect is one period.
We also refer to [ ? ] formore discussions about the identification of the length of carryover effects as well as a data drivenstrategy to select a plausible carryover effect.
In the remaining of this paper, we use the short-hand notation Y t ( w t − 1: t ) =
Y t ( w 1: T ) to focus onthe dependence of the potential outcomes at two consecutive periods.
Using this short-hand notation,the observed outcomes are related to their respective potential outcomes as follows,
[Equation]
2.3 The causal effect
The primary focus of this paper is to understand the time-averaged Total Treatment Effect , i.e., thedifference between the average outcomes when the aggregated unit is exposed to treatment at alltime periods, relative to when it is exposed to control at all time periods [6].
Mathematically, this isdefined as
[Equation]
where 1 and 0 are two assignment paths with all treatments w 1: T = 1 1: T and controls w 1: T = 0 1: T ,respectively.
Based on Assumption 1, they can be simplified as w t − 1: t = 1 t − 1: t and w t − 1: t = 0 t − 1: t 82 accordingly.
The causal effect as defined in (1) captures the effect of permanently deploying a new policy.
In the
Lyft example [7], this could reflect the change in average ride-making rates if the company rolls out anew subsidy policy across all local regions in a market.
Such a causal estimands directly helps datascientists evaluate the benefits of permanently deploying such a policy.
Since the causal effect τ ( Y )is never directly observable, our goal is to estimate the casual effect using observations from the timeseries experiments efficiently, which requires a careful design of experiments.
Problem formulation
As introduced in Section 2.1, in this paper we focus on a design-based perspective of experimentaldesign, and rely on variations introduced by the random assignment path for doing inference.
In thissection we study, as a decision-making problem, the design of time series experiments that governsthe selection of the random assignment path to gain high statistical power for inference.
Formally, the design of an experiment is a discrete probability distribution η ( · ) : { 0 , 1 } T → [0 , 1]over the assignment paths, such that,
[Equation]
In practice, the experimenter would sample an assignment path W 1: T = w 1: T from the distributionη ( · ) , and then implements this assignment path to conduct an experiment.
Once the experimenthas been conducted, the experimenter collects the observed outcomes { Y t } t ∈ [ T ] , and uses both therealized assignment path and the observed outcomes to estimate the causal effect.
A commonly used estimator is the Inverse Propensity Weighted (IPW) estimator, which is alsoreferred to as the Horvitz-Thompson estimator [15], as follows,
[Equation]
It is a well-known result that the IPW estimator is unbiased [5], i.e., E η [ (cid:98)
τ ( Y , η, w )]
= τ ( Y ) .
To evaluate the quality of a specific η ( · ) , we adopt the decision-theoretic framework [3, 4] andfocus on the variance of the IPW estimator, Var η
( (cid:98) τ ( Y , η, w )) =
E η [( (cid:98) τ
( Y , η, w ) − τ ( Y )) 2 ] .
Sincethe IPW estimator is unbiased, the variance of the estimator is equivalent to the risk function, orthe mean squared error of the estimator.
Since the variance of the IPW estimator depends on thepotential outcomes Y , there may not exist one design of experiment that uniformly achieves a smallvariance in all scenarios.
To determine the design of experiment, we adopt the minimax decision rule[3, 30, 21, 6] and find the design of experiment that minimizes the worst-case variance against anadversarial selection of potential outcomes, i.e.,
[Equation]
One compelling reason of adopting the minimax decision rule is that we do not impose any parametricor structural models on the potential outcomes.
Yet to make the decision making problem feasible,and to lay the foundation for inference, we impose a bounded support assumption of the potentialoutcomes.
Assumption 2 (Bounded potential outcomes) .
There exists B > 0 , such that for any t ∈ [ T ] , w 1: T ∈{ 0 , 1 } T , Y t ( w 1: t ) ∈ [0 , B ] .
Equivalently, Y = [0 , B ] T .
Assumption 2 is typically satisfied in practice as it assumes that the set of potential outcomes isnon-negative and upper bounded by some possibly large constant.
Note that this assumption issatisfied even if the potential outcomes arise from a stochastic process, as long as the process doesnot have a point mass at infinity.
As we will show in the next section, our experimental design doesnot require the knowledge of the upper bound B , just its existence.
The design of time series experiments
4.1 A balanced design
In this section, we study the design of time series experiments.
Solving the optimization problemdefined by (3) is typically challenging, as the inner problem is a maximization over a convex functionwhose optimal solution lies on the exponentially many extreme points of the polyhedron Y , andthe outer problem is hindered by the challenges of evaluating the inverse probabilities in the IPWestimator.
Even if the optimization problem can be numerically solved, inference and testing typicallyrequire a clear understanding of the randomization mechanism of the design, which a numericalsolution fails to provide.
Furthermore, from a practical perspective, the randomization mechanismneeds to be easy to implement when the design of the experiment is brought to a company.
Instead of finding the exact optimal solution, we propose a balanced design that has a sub-optimaltheoretical performance and is easy to implement in practice.
Suppose T is odd.
Consider thefollowing balanced design η † of the time series experiments:
During the first T − 1 periods, conduct a complete randomization with a equal number of
treatments and controls.
That is, T − 1 (cid:80)
[Equation]
Set the treatment assignment of the last period to be the same as that of the first period, i.e.,
W T =
W 1 .
The first step reflects the key idea of balancing inspired by the classical completely randomizeddesign.
The second step simply deals with the boundary conditions of the design.
Because W T = W 1 ,we could specify the precedent period of t =
T to be t = 2 , i.e., W T
+1 =
W 2 .
This simplifies theanalysis later a lot without making the design significantly different.
We illustrate in Figure 1 the example of two assignment path realizations with T = 9 .
This design the design
T = extends the classical completely randomized experiment to the temporal interference setup.
A classical completely randomized experiment [10, 22] fixes the number of treatments and controls in advance before randomization is applied, ensuring that the number of valid observations under treatment and control is equal to the number of units who receive treatment and control, respectively.
However, under the temporal interference setup that we consider, whether Y t ( w t − 1: t ) is a valid observation at period t depends on the treatment assignments during both periods t − 1 and t (e.g., Yunder the assignment path 1 in Figure 1 is not a valid observation).
This suggests a careful analysis for the underlying dependence of the design.
To this end, we first introduce some statistics of interest which are involved in our analysis.
Define
[Equation]
to be the average outcomes of treatment and control, respectively.
Given these, we further define
[Equation]
to be the variance of the average outcomes at two consecutive periods.
Similarly, we can also measure the variance of the treatment effect using
[Equation]
These three statistics help extend the analysis of classical complete randomization by bridging theoutcomes at two consecutive periods explicitly.
We have the following theorem that characterizes thevariance of our balanced design.
Theorem 1.
For any potential outcomes Y , the variance of the balanced design η † can be decomposed as
[Equation]
We prove Theorem 1 in Appendix A.1.
Later in Section 5, we will further develop a variance upperbound and a corresponding estimate for doing inference.
4.2 Performance analysis
Let Var η ∗ ( (cid:98) τ ) denote the worst-case variance of an optimal design η ∗ which solves (3) exactly.
Inthis section, we analyze the performance of the balanced design η † by first comparing its worst-casevariance Var η † ( (cid:98) τ ) to Var η ∗ ( (cid:98) τ ) , the worst-case variance of the optimal design.
The following result shows that the worst-case outcomes against the balanced design
η † can becharacterized explicitly, as can the worst-case variance.
Theorem 2.
The worst-case outcomes against the balanced design η † can be written as
[Equation]
This leads to the worst-case variance
[Equation]
We prove Theorem 2 in Appendix A.2 and outline here the key ideas of the proof.
The proof showsthat, given any other potential outcomes, one can always transform the outcomes to the ones thatmatch the above structure, obtaining no smaller variance.
Since solving (3) for obtaining the worst-case variance of the optimal design Var η ∗ ( (cid:98) τ ) is difficult,we then construct its lower bound as follows.
Theorem 3.
The worst-case variance of the optimal design Var η ∗ ( (cid:98) τ ) has a lower bound
[Equation]
We prove Theorem 3 in Appendix A.3.
This lower bound directly implies a 2-approximation ratio forthe balanced design.
Proposition 4.
The balanced design η † has an approximation ratio
[Equation]
We further compare the balanced design η † to another Bernoulli design studied in [ ? ].
The optimal
Bernoulli design η # of time series experiments was developed, where the experimenter can onlydraw i.i.d.
Bernoulli trials at some time periods to determine the random assignments.
Theorem 2 in[6] shows that the optimal Bernoulli design η # has the worst-case variance Var η # ( (cid:98) τ ) =
16 T − 56 ( T − 1) 2 B 2 .
Proposition 5.
The relative performance of the worst-case variances between the balanced designη † and the optimal Bernoulli design η # is given by
[Equation]
4.3 Simulation study
In this section, we conduct a simulation study to investigate the general performance of our balanceddesign η † .
First of all, we set the outcomes Y to follow the worst-case structure in (2).
For eachnumerical experiment, we randomly sample an assignment path, compute the Horvitz-Thompsonestimator (2), and repeat the procedure 10000 times to estimate the performance of the design.
In
Figure 2, the variance of the balanced design is significantly lower than that of the optimal Bernoullidesign.
Note that both designs are evaluated under the outcomes in (2), which correspond to theworst-case scenario for the balanced design, but do not correspond to the worst-case scenario for the
Bernoulli design.
This further justifies the robustness of the balanced design.
[Figure] [Figure]
Next, we consider the following outcome model that is studied in [6]:
[Equation]
(7) where ϵ t ∼ N (0 , 1) .
Here α t depicts the base structure of the time series, β 0 governs the directcausal effect of the treatment, and β 1 governs the carryover effect of the treatment.
The causal effectof interests is τ = β 0 + β 1 .
We first let α t = log( t ) , β 0 = 0 .
5 , β 1 = 0 .
In Figure 3, the balanceddesign dominates the Bernoulli design.
Our new design is an order of magnitude better than theprevious Bernoulli style design; for example, the balanced design with T = 21 has a lower variancethan the Bernoulli design with T = 201 .
This implies that the benefit of the balanced design could bemore significant beyond the worst-case scenario.
Moreover, we test different outcome models bychanging the parameters in (7) and layout the corresponding variances in Table 1.
[Table]
To summarize, in this section, we propose a balanced design of time series experiments and studythe performance of its variance both in the worse-case perspective analytically, and in a generalperspective numerically.
This type of design is simple in nature, easy to implement in practice, andmore importantly, effective for making inference.
To see this, in the next section, we will investigatehow the variance reduction is translated to the value for the inference.
Inference and testing
After running an experiment, we want to test whether the estimate treatment effect is systematic ordue to change.
To do that, we consider the following null hypothesis for the time-averaged totaltreatment effect and the alternative:
[Equation]
To test this null hypothesis, we first derive a finite population central limit theorem to approximatethe distribution of the Horvitz-Thompson estimator under our design, together with a conservativevariance estimation.
We then conduct simulations to examine the effectiveness of inference usingdifferent designs.
5.1 Central limit theorem
Before stating our central limit theorem, we must make an assumptions that guarantees that thevariance is not dominated by a small number of time periods.
Assumption 3 (Non-negligible variance) .
[Equation]
This assumption often holds in practice and is regularly made by researchers [6].
Theorem 6.
Under Assumptions 1 - 3, the limiting distribution of the Horvitz-Thompson estimatorhas an asymptotic normal distribution.
That is, as T → + ∞ ,
[Equation]
We prove Theorem 6 in Appendix A.4.
The balanced design studied in this paper is inspired bythe classical complete randomization, where the number of valid observations for treatment andcontrol is fixed and known before the randomization.
A generalized central limit theorem hasbeen developed [22] to handle multiple treatments and multi-dimension outcomes.
However, thenumber of valid observations for treatment and control becomes random under temporal interference,which requires more complex and careful analysis.
We adopt the framework of the permutationstatistics [8, 13, 31] to analyze the fundamental behavior of the random permutation in the balanceddesign.
Whenever we adopt design-based inference, the variance of the Horvitz-Thompson estimator dependson the potential outcomes of both treatment Y t ( 1 ) and control Y t ( 0 ) at all periods t ∈
[ T ] , as shownin (4).
Therefore, to use the normal approximation for testing, we need to replace the unknown truevariance Var η † (ˆ τ ) by some variance estimate.
Although one can always resort to the worst-casevariance (2) as a conservative estimate, the observations from the experiment are not well leveraged.
We aim to develop a more informative variance estimate ˆ σ 2U .
To this end, we introduce several sampleestimates.
First we define
[Equation]
to be the sample estimate for Y ( 1 ) and Y ( 0 ) .
Next we define
[Equation]
to be the sample estimate for S t and S c .
Proposition 7.
There exists an upper bound for the variance of the balanced design.
That is,
[Equation]
This upper bound has an unbiased estimate
[Equation]
We prove Proposition 7 in Appendix A.5.
Compared to the true variance (4), the upper bound is nottight in general.
Nevertheless, we will show that it admits a great inference power for testing the nullhypothesis.
5.2 Simulation study
We do the simulation study using the same outcome model in (7) with α t
= log( t ) .
We aim tojustify the normal approximation and test the effectiveness of the inference.
Following the simulationprocedure in Section 4.3, we not only calculate the Horvitz-Thompson estimator (2) based on theobserved outcomes, but also calculate the conservative variance estimate (11).
5.2.1 Asymptotic normality
We first justify the normal approximation using the samples generated from the outcome model withβ
= β 1 = 1 and T = 401 .
More specifically, we generate 100000 samples of the estimator (cid:98) τand conduct a Kolmogorov–Smirnov test [28] for the null hypothesis that the samples come froma normal distribution.
The test returns an estimated p-value 0.57, which implies a good normalapproximation.
Figure 4 shows the histogram and the Q-Q plot that correspond to the distributioninduced by (cid:98)
τ − τ √ Var η ‡ ( (cid:98) τ | Y ) , for which we numerically compute Var η ‡ ( (cid:98)
τ | Y ) using samples from thesimulation.
D e n s i t y        4 Theoretical Quantiles
S a
[Figure]
p l e
Q u a n t il e s
5.2.2 Rejection rate
We test the null hypothesis (8) using the normal approximation.
We plug in the conservative varianceestimate to obtain the estimated p-value ˆ p .
We reject the null hypothesis if ˆ p < 0 .
By repeatingthis procedure 10000 times, we summarize the frequency of a null hypothesis being rejected(i.e.rejection rate).
We present the rejection rates as the number of periods T grows under three outcome models in Figure5.
The balanced design leads to the right decision more efficiently than the optimal Bernoulli design in all scenarios.
In particular, when there exists some degree of the casual effect τ = β 0 + β 1 > 0 , thebalanced design only needs 20% as much time periods as in the optimal Bernoulli design to achievethe same rejection rate.
Furthermore, we plot the average point estimates and the average confidence intervals in Figure 6.
Wecan observe that all the estimates are indeed unbiased and the balanced design consistently achievesmuch narrower confidence intervals.
Specifically, for getting the average confidence interval above 0,the balanced design only needs 350 periods when τ = 1 and less than 100 periods when τ = 2 .
Incontrast, the optimal Bernoulli needs more than 500 periods in both scenarios.
This further justifiesthat the balanced design is more data-efficient
[Figure] [Figure]
Concluding remarks
This paper studied the design of time series experiments in the presence of interference.
The proposedbalanced design improves the casual estimator’s variance leading to better inference and testingefficiency.
The main results could also be extended to the design of panel experiments, where weconduct time series experiments on multiple units simultaneously.
A Proof of main results
A.1 Proof of Theorem
Proof.
Proof of Theorem 1.
We first compute several joint probabilities that will be used later.
Thepropensity score
[Equation]
T − 8 , where (cid:0) nk (cid:1) stands for the combinatorial number of choosing k items from a total of n items.
Similarly,the probabilities that we can observe three and four consecutive treatment/control are
Pr
[Equation]
, and
Pr( W t
[Equation]
: t +2 = 1 ) =
Pr
[Equation]
W
[Equation]
2 = 0 )
=
[Equation] [Equation]
[Equation]
(4 T
[Equation]
8)(2 T
[Equation]
6)(2 T − 8) .
Furthermore, we have
Pr( W t − 1: t
[Equation]
1 , W t +
[Equation]
t +2 = 0 )
=
Pr( W t − 1: t
[Equation]
0 , W t +1: t +2 = 1 ) =
[Equation]
[Equation]
(cid:1) (cid:0) T − 1 ( T −
[Equation] [Equation]
(cid:1)
[Equation]
( T −
[Equation]
( T −
[Equation]
( T −
[Equation]
(4 T − 8)(2 T − 6)(2 T − 8) .
Now we are ready to analyze the variance.
We first write the estimator as follows,
[Equation]
The variance of the estimator can be decomposed as
[Equation]
We first examine the first part:
Var
[Equation]
Let us define δ t,t ′ to be the distance between two periods.
Using the joint probabilities we havederived, we can specify the covariance in three scenarios:
[Equation]
Therefore, we can calculate the variance as follows,
[Equation]
Var
[Equation] [Equation]
−
[Equation]
Y
[Equation]
Y
[Equation]
Because of
T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 )) 2 =
T (cid:88) t =
Y 2 t ( 1 ) −
T −
T (cid:88) t = T (cid:88) t ′ =
Y t ( 1 ) Y t ′ ( 1 ) ,
T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 )) ( Y t +1 ( 1 ) − Y ( 1 )) =
T (cid:88) t =
Y t ( 1 ) Y t +1 ( 1 ) −
T −
T (cid:88) t = T (cid:88) t ′ =
Y t ( 1 ) Y t ′ ( 1 ) the variance is equivalent to
Var (cid:32)
[Equation]
Pr
[Equation]
(cid:33)
[Equation]
( T
[Equation]
1)( T
[Equation]
)( T
[Equation] [Equation]
T (cid:88) t
[Equation]
( Y t ( 1 )
[Equation]
Y ( 1 ))
+
[Equation] [Equation]
[Equation]
( T − 1)( T − 4)( T − 3)
T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 )) ( Y t +1 ( 1 ) − Y ( 1 ))
+ ( T − 1)( T − 4)
T (cid:88) t =
Y 2 t ( 1 ) − ( T − 1)( T − 4)( T − 3)
T (cid:88) t =
Y t ( 1 ) Y t +1 ( 1 ) .
Similarly, we can characterize the variance for outcomes of control:
[Equation]
Next
[Equation]
we examine the covariance between the outcomes of treatment and control:
[Equation]
The product of expectations is given by
[Equation]
Using the jointly probabilities we have derived again
[Equation]
we can specify the expectation of the productin three scenarios:
[Equation]
Therefore
[Equation]
we can calculate the covariance as follows,
[Equation]
t = 0 ) (cid:33) =
Y ( 1 ) Y ( 0 ) − ( T − 2)( T − 1) ( T − 3)( T − 4)( T − 1)
T (cid:88) t = (cid:88) δ t,t ′ ≥
Y t ( 1 ) Y t ′ ( 0 )
Because of
T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 ))
( Y t ( 0 ) − Y ( 0 )) =
T (cid:88) t =
Y t ( 1 ) Y t ( 0 ) −
T −
T (cid:88) t = T (cid:88) t ′ =
Y t ( 1 ) Y t ′ ( 0 ) ,
T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 ))
( Y t +1 ( 0 ) − Y ( 0 )) =
T (cid:88) t =
Y t ( 1 ) Y t +1 ( 0 ) −
T −
T (cid:88) t = T (cid:88) t ′ =
Y t ( 1 ) Y t ′ ( 0 ) the covariance is equivalent to
[Equation]
Now we are going to reformulate the covariance using previous expressions.
Since we have thefollowing two equations
[Equation]
( 1 ) − Y t +1 ( 0 ) − Y ( 1 ) + Y ( 0 )) and
T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 ))
( Y t ( 0 ) − Y ( 0 )) =
T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 )) 2 +
T (cid:88) t
= ( Y t ( 0 ) − Y ( 0 ))−
T (cid:88) t
= ( Y t ( 1 ) − Y t ( 0 ) − Y ( 1 ) + Y ( 0 ))
2 , we then rewrite the covariance
as
T −2( T − 4)( T − 3)( T − 1) (cid:32) T (cid:88) t =
( Y t ( 1 ) − Y ( 1 )) 2 +
T (cid:88) t
= ( Y t ( 0 ) − Y ( 0 )) 2 −
T (cid:88) t
= ( Y t ( 1 ) − Y t ( 0 ) − Y ( 1 ) + Y ( 0 ))(cid:33) + 2 T −
( T − 4)( T − 3)( T − 1) (cid:32) T (cid:88) t
= ( Y t ( 1 ) − Y ( 1 ))
( Y t +1 ( 1 ) − Y ( 1 ))
+
T (cid:88) t
= ( Y t ( 0 ) − Y ( 0 ))
( Y t +1 ( 0 ) − Y ( 0 ))
−
T (cid:88) t
= ( Y t ( 1 ) − Y t ( 0 ) − Y ( 1 ) + Y ( 0 ))
( Y t +1 ( 1 ) − Y t +1 ( 0 ) − Y ( 1 ) + Y ( 0 )) (cid:33) − ( T − 4)( T − 1)
T (cid:88) t =
Y t ( 1 ) Y t ( 0 ) + 2( T −
[Equation]
Finally, putting all parts (12), (13), (14) together and using S c , S t , S τ , we derive the variance of theestimator
[Equation]
This can be further simplified as
[Equation]
A.2 Proof of Theorem
Lemma 8.
The worst-case outcomes must satisfy the following structure:
[Equation]
Proof.
Proof of Lemma 8.
We first expand the variance by definition and have the following:
[Equation]
This is a quadratic function with variables Y t ( 1 ) , Y t ( 0 ) , ∀ t ∈ { 2 , 3 , . . .
, T } .
To show it is also convex,we can rewrite the summation as y ′ Σ y , where y ∈ R 2( T − 1) is the vector of all variables and Σ isa symmetric matrix of coefficients.
Because variance is non-negative, we know that y ′ Σ y ≥
forany y , which implies that Σ is PSD and the function is convex in y .
Since the inner optimization isa minimization in a bounded feasible region, the worst-case solution can be attained at one of theextreme points.
That is,
[Equation]
Next, given any outcomes at the extreme point, we will argue that transforming into the structure(16) leads to a larger variance.
To see this, we need to carefully analyze the coefficients of Σ .
Weintroduce some shorthand notations for future reference.
Due to the symmetry of the design, we set
[Equation]
Based on the joint probabilities we have derived in the proof of Theorem 1, the coefficients of Σ canbe characterized according to δ t,t ′ :
[Equation]
It is easy to see that both q + ( δ t,t ′ ) and q − ( δ t,t ′ ) are decreasing in δ t,t ′ .
Therefore, the closer twooutcomes B , the more they contribute to the variance.
Suppose we are given some outcomes at theextreme point and there are T 1 periods whose outcome of treatment is B while T 0 periods whoseoutcome of control is B .
W.L.O.G., assuming T 1 ≥ T 0 , let us consider the following alternativeoutcomes:
[Equation]
We check the variance of the alternative outcomes using the monotonicity of q + ( δ t,t ′ ) and q − ( δ t,t ′ ) .
Since the alternative outcomes group B together with the minimal distance, the variance from theoutcomes of treatment(control) increases.
Moreover, because the alternative outcomes synchronizethe outcomes between treatment and control as much as possible, the covariance from the outcomesbetween treatment and control increases as well.
Together, the alternative outcomes achieve a largervariance.
Lastly, it remains to show that further transforming (17) into (16) gives us a larger variance.
Essentially, the transformation is doing
[Equation]
This can be illustrated using the examples in Figure 7 with T = 9 .
To see that the variance increases,
394 due to the symmetry of the design, we need to show that the covariance between blue outcomes andred outcomes is getting larger.
Precisely,
[Equation]
It is sufficient to show that
[Equation]
Because of the monotonicity of q + ( · ) and q − ( · ) , it suffices to show that
[Equation]
( T
[Equation]
− T
[Equation]
δ t
[Equation]
t ′
[Equation]
q
[Equation]
( δ t
[Equation]
t ′ )
[Equation]
Plugging in the expressions, this is equivalent to
[Equation]
Since T 1 − T 0 is bounded above by T − 1 2 , it suffices to show that
[Equation]
Hence, the worst-case outcomes must obey the structure that:
[Equation]
Proof.
Proof of Theorem 2 From Lemma 8, we know that the worst-case outcomes obey the followingstructure
[Equation]
We need to show that s = T +12 .
We do it by contradiction.
Suppose that s < T +12 , then we can set Y s +1 ( 1 )
=
Y s +1 ( 0 ) =
B .
In this way, we have one morepair of outcomes B which contributes to the variance.
To show that the variance increases, it isequivalent to prove that
[Equation]
Since q + ( δ t,t ′ ) + q − ( δ t,t ′ ) takes negative value when δ t,t ′ ≥ 2 , it is sufficient to show that
[Equation]
Plugging in the expressions of q + and q − , we have
[Equation]
In the other way around when s > T +1 2 , then we can set Y s − 1 ( 1 ) =
Y s − 1 ( 0 ) = 0 .
Following thesimilar argument, it is sufficient to show that
[Equation]
Plugging in the expressions again, we have
[Equation]
Hence, the variance reaches the maximum when s = T +12 .
To further derive the worst-case variance, we can simply use the variance decomposition (15).
Notethat the last three parts are all zero, the worst-case variance can be calculated by
[Equation]
A.3 Proof of Theorem 3
Lemma 9.
Let us consider two consecutive time periods t and t + 1 .
For any symmetric design, wehave the following inequality:
[Equation]
Proof.
Proof of Lemma 9.
We first reformulate the inequality as
[Equation]
Now let us focus on three time periods: t − 1 , t and t + 1 .
There are 8 possible assignment paths.
We layout 4 of them and the remaining ones are just symmetric:
[Equation]
with their probability mass denoted as a 3 , a 2 , 1 , a 2 , 2 , a 2 , 3 respectively.
Then we can characterize theprobabilities in the inequality using these a :
[Equation]
Since a 3 + a 2 , 1 + a 2 , 2 + a 2 , 3 = 0 .
5 , it is equivalent to show
[Equation]
Notice that a 2 , 3 only appears on the left-hand-side, so it is sufficient to show
[Equation]
which can be further simplified as
[Equation]
This is true for any a .
Proof.
Proof of Theorem 3.
In the proof of Lemma 8, we rewrite the variance by introducingy ∈ R 2( T − 1) to denote the vector of outcomes.
Our original minimax problem is equivalent to
[Equation]
where Σ is some covariance matrix that can be mapped from a feasible design and the adversaryfinds an outcome vector to maximize the variance.
To get a lower bound of the optimal worst-casevariance, we consider a randomized feasible solution ˜
y
regardless of the covariance matrix in theouter optimization.
We first combine every two time periods as a group, so we have overall n = T − 1 2groups.
We then randomly pick half of the groups and set their corresponding outcomes to B and others to 0 .
Let h ( i ) denote the group of some outcome ˜
y i .
In this way, for any two outcomes ˜ y iand ˜
y j from the same group(i.e.
h ( i ) =
h ( j ) ), E [˜ y i ˜ y
j ] = 12 B 2 ; for any two outcomes from differentgroups, E [˜ y i ˜ y
j ] = n − 2 4( n − 1) B 2 .
Now we can bound the inner optimization as follows:
[Equation]
Note that the last term is non-negative, so it implies that
[Equation]
Then it remains to investigate Σ i,j when two outcomes ˜
y
i and ˜ y
j are from the same group(i.e.
two consecutive periods).
Let us focus on what will happen in one group.
First of all, it is easy to observe that the following is true for any design:
[Equation]
We have 4 such pairs in one group, so they contribute
[Equation]
1) B 2 to the variance.
Next, if we set the assignments in the above equation to be jointly 0 or 1 , we are not able to know the exact values.
Fortunately, based on Lemma 9, we can still bound the variance to which they contribute by
[Equation]
Lastly, as we have n groups, we get the lower bound
[Equation]
Together with the upper bound, the approximation ratio follows easily.
A.4 Proof of Theorem
Proof.
Proof of Theorem 6.
Let us first define
[Equation]
Let π be a random permutation that shuffles the original indices:
[Equation]
Given these, we can rewrite the estimator as
[Equation]
(21) where T (cid:80) t ̸ = t ′ indicates
T (cid:80)
[Equation]
t ′ =2: t ̸ = t ′ .
To derive the normal approximation of this, we adopt Stein’smethod of exchange pairs for double-index permutation statistics proposed in [24].
Specifically, theyconstruct an exchangeable pair as follows.
Let t and t ′ be distributed uniformly over 1 , . .
, T − 1conditioned that t ̸ = t ′ .
Define the permutation π ′
=
( π ( t ) π ( t ′ ))
◦ π so that π ′ is the permutationwhere π ′ ( s )
=
π ( s ) for all k ̸ = t, t ′ , and where π ′ ( t ) =
π ( t ′ ) and π ′ ( t ′ )
=
π ( t ) .
Let V
= (cid:98) τ , andwe define the other two random variables for proof purposes:
[Equation]
Then we have V ′ =
( V ′ 1 , V ′ 2 , V ′ 3 ) =
V ( π ′ ) to be the estimators with the exchange pair.
For therandom exchange pair ( t, t ′ ) , we have the following equations:
[Equation]
They further satisfy that
[Equation]
where
[Equation]
To be self-contained, we re-state the following theorem to show the asymptotic normality.
Theorem 2 in [24].
Assume that ( V , V ′ ) is an exchangeable pair of random vectors such that
[Equation]
with Σ ∈ R 3 × 3 symmetric and positive definite.
If (22) holds and Z has a 3-dimensional standardnormal distribution, we have for every three times differentiable function h ,
[Equation]
where
[Equation]
To apply the theorem, we first note that E [ V ] = 0 may not hold.
Nevertheless, we can simplyde-mean V by E [ V ] , and thus the condition is satisfied.
Next, it is easy to see γ
=
O ( T ) and weneed to characterize A, B, C using (20):
• A: Let us use the analysis of Var E V ( V ′ 1 − V 1 ) 2 as instance.
First of all, we have
[Equation]
Let π ′ be the permutation with the exchange pair t, t ′ and π ′′ be the permutation with theexchange pair s, s ′ .
To analyze the variance of E V ( V ′ 1 − V 1 ) 2 , it suffices to see that
[Equation]
This further leads to
[Equation]
Following the same procedure, we can obtain that
[Equation]
• B: Let us use the analysis of E [ | ( V ′ 1 − V 1 ) 3 | ] as instance.
We take the conditioning on theexchange pair ( t, t ′ ) , which gives
[Equation]
Following the same procedure, we can obtain that E (cid:12)(cid:12) ( V ′ i − V i ) (cid:0) V ′
j − V j (cid:1) ( V ′ k − V k ) (cid:12)(cid:12) =
O (cid:18)
T(cid:19) .•
C:
Since R 2 = R 3 = 0 , we simply need to consider R 1 .
[Equation]
(cid:118)(cid:117)(cid:117)(cid:117)(cid:116)
[Equation]
Putting A, B, C together, we have
[Equation]
Note that Σ 1 / 2 1 , 1 = (cid:112) Var η † ( (cid:98) τ ) has an order of 1 √ T .
If we normalize V 1 by the standard deviation, thisleads to the typical rate of convergence O (cid:16) 1 √ T (cid:17) for asymptotic normality.
A.5 Proof of Proposition 7
Proof.
Proof of Proposition 7 We write the original variance decomposition (4):
Var( (cid:98) τ | Y )
=
8( T − 2)( T − 3) 2 ( T − 1) (cid:0) S t + S c (cid:1) − (4 T − 10)( T − 2) ( T − 3) ( T − 4)( T − 1) S
τ
+ ( T − 1)( T − 4) T (cid:88) t = ( Y t ( 1 ) − Y t ( 0 )) 2 − ( T − 1)( T − 3)( T − 4)
T (cid:88) t
= ( Y t ( 1 ) − Y t ( 0 ))
( Y t +1 ( 1 ) − Y t +1 ( 0 )) .
First, because of
T (cid:88) t
= ( Y t ( 1 ) − Y t ( 0 ))
+ ( Y t ( 1 ) − Y t ( 0 ))
( Y t +1 ( 1 ) − Y t +1 ( 0 ))
{+= 
+}
T (cid:88) t
= ( Y t ( 1 ) + Y t +1 ( 1 ) − Y t ( 0 ) − Y t +1 ( 0 )) 2 , we rewrite the variance as
Var( (cid:98) τ | Y )
=
8( T − 2)( T − 3) 2 ( T − 1) (cid:0) S t + S c
(cid:1) − (4 T − 10)( T − 2) ( T − 3)
( T − 4)( T − 1) S
τ
+
T −( T − 1)( T − 3)( T − 4)
T (cid:88) t
= ( Y t ( 1 ) − Y t ( 0 ))
− 2( T − 1)( T − 3)( T − 4)
T (cid:88) t
= ( Y t ( 1 ) + Y t +1 ( 1 ) − Y t ( 0 )
[Equation]
Removing the non-positive parts gives
[Equation]
Next, because of the non-negative outcomes, we have
[Equation]
This finally leads to the upper bound
[Equation]
Following the similar argument in [17], we obtain an unbiased estimate for the upper bound
[Equation]
where ˆ S t , ˆ S c are the sample estimates and Y t is the observed outcome.