Privacy Aware Experiments without Cookies
ABSTRACT
Consider two brands that want to jointly test alternate web experiences for their customers with an A/B test.
Such collaborative tests are today enabled using third-party cookies , where each brand has information on the identity of visitors to another website.
With the imminent elimination of third-party cookies, such A/B tests will become untenable.
We propose a two-stage experimental design, where the two brands only need to agree on high-level aggregate parameters of the experiment to test the alternate experiences.
Our design respects the privacy of customers.
We propose an estimater of the Average Treatment Effect (ATE), show that it is unbiased and theoretically compute its variance.
Our demonstration describes how a marketer for a brand can design such an experiment and analyze the results.
On real and simulated data, we show that the approach provides valid estimate of the ATE with low variance and is robust to the proportion of visitors overlapping across the brands.
CCS CONCEPTS • Applied computing → Marketing ; Electronic commerce ; • Security and privacy → Privacy protections
ACM Reference Format: Shiv Shankar, Ritwik Sinha, Saayan Mitra, Moumita Sinha, Viswanathan
Swaminathan, and Sridhar Mahadevan.
Privacy Aware Experiments without Cookies.
In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (WSDM ’23).
ACM, New York, NY, USA, 8 pages.
https://doi.org/XXXXXXX.XXXXXXX
INTRODUCTION
A/B testing (randomized experiment) is the gold standard for optimizing customer experiences on the web.
Cookies have historically played an important role in ensuring that the same customer receives a consistent (or sticky ) experience, thus ensuring the validity of the A/B test.
Consider the A/B testing scenarios, where different brands are collectively testing alternate customer experiences.
Some examples include, brands under a hotel chain, different franchises
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than ACM mustbehonored.
Abstractingwithcreditispermitted.
Tocopyotherwise,orrepublish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Request permissions from permissions@acm.org.
WSDM ’23, Feb 27– March 03, 2023, Singapore © 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/18/06...
$15.00 https://doi.org/XXXXXXX.XXXXXXX under a sporting league, related clothing brands jointly owned by a holding company, or multiple departments of a government.
Thus far, such A/B tests have relied on the third-party cookie , which requires brand 1’s cookie being available on brand 2’s website.
This is the only mechanism to ensure that the same individual receives the same experience across both web properties.
A recent trend on the web has been the increased focus customer privacy.
This is embodied in laws like the General Data Protection
Regulation (GDPR) 1 , which require an explicit permission from the visitor for tracking web sessions.
A second manifestation of this trend is that all major browser ecosystems have taken steps to discontinue third-party cookies [ ? ].
This presents significant challenges for brands that want to jointly optimize customer experiences [18].
Formally, this is our question of interest.
Consider a binary treatment deployed across two websites, for example, Brand 1 and Brand 2 have jointly decided to optimize the experiences on their websites (e.g., the discount offer, color of the buy button, layout of the website, and so on).
Let’s call the treatments 1 and 2.
A user may visit one or both websites.
In the ideal case a user who was assigned to the group which experiences treatment 1 on the first website, will also see treatment 1 on the second website.
However, in the case where the identity of the user cannot be confirmed across the two websites (due to a lack of third-party cookies), the user is not guaranteed to see the same treatment across both websites.
This leads to four potential treatment exposures, namely 𝑌 11 , 𝑌 21 , 𝑌 12 , and 𝑌 22 (first and second positions denoting the treatments at the first and second websites).
The true treatments of potential outcomes of interest are 𝑌 11 and 𝑌 22 .
The question of interest is whether the treatment has any effect; which in the world with cookies corresponds to the effect 𝐸 [ 𝑌 11 − 𝑌 22 ] .
This problem is easily solved if the websites could share information about which users were exposed to which treatment; while retaining independent decision/allocation process.
However, it is not clear whether this effect can be estimated without sharing of such individual user level information.
In our work we present a multi-stage randomization design which allows the estimation of the desired effect without sharing any user-level information.
Figure 1 describes our proposed design.
The two brands only agree on two aggregate parameters of the test: 1) the two treatments being tested, (2) a notion of clusters (divide the population into 𝐶 1 and 𝐶 2, e.g., geographies, time, device types), and (3) the nuisance parameter 𝛼 ( ≠ 1 / 2).
No individual or identity data needs to be shared between the brands.
First, we show that our proposed average treatment effect (ATE) is unbiased and compute its variance theoretically.
Next, we further propose a way of performing regression adjustment, which further helps statistical power by using attributes of the web visitors.
Finally, on experiments in simulated and real world web experiment data, we show that our estimate has lower bias than the naive ATE (difference of treatment means).
In our demo, we show how a marketer for a brand can use our design a two-stage test and analyze the results.
RELATED WORK
The problems of using cookie-level identifiers on the web as as a proxy for the individual’s true identity is well studied [2, 4, 8, 15, 22].
But looking beyond cookies as the identity in digital marketing is less well studied [26].
The Privacy Sandbox [3] is an initiative with proposals tackling privacy related challenges for ad-targeting, delivery and measurement.
These include proposals like FLoC and
TOPICS, but they are all early proposals subject to changes and uncertainty.
Resolving the user’s identity can overcome most measurement and attribution [24, 25] issues, and considerable research has gone into stitching fragmented user behaviour [6, 10, 12, 23].
But these strategies rely on using generic features such as IP to represent the same human user, or use more detailed but private data.
The first persists privacy concerns and leads to inaccurate stitching, while the second relies on first party data from walled gardens and precludes any cross-ecosystem analysis.
Industrial consortiums like “The TradeDesk Unified ID 2.0” and “Advertising ID Consortium”, are building their own unique identifiers based on people-based identifiers (Email ID, Device).
Some firms such as Zillow also create custom segments from their first-party data and allow advertisers access.
However concerns about scalability and possible future regulation, means this is not sustainable long term strategy.
As such Kamena [11] propose for greater use of media mix modeling as a complementary approach to user-level attribution.
Interference related problems have been well studied in the literature [1, 9].
But these assume strong restrictions on the structure of spillover.
Recently some work has focused on how to account for general interference [17, 19, 27] .
However all of these methods rely on complete knowledge of interference structure which is not possible with our setup.
Another problem related to disappearance of cookies is that of identity fragmentation [4, 16].
Lin and Misra [14] demonstrate treatment effect attenuation in presence of identity fragmentation.
Coey and Bailey [5] also provide a debiasing estimator for cookie-level estimates under similar assumptions.
But both of these primarily target he loss of first-party cookies.
Our approach is instead based on conducting parallel experimentation followed by stratified aggregation.
Instead of constructing user-level records, this approach works by running multiple experimentation at stratified cohorts and constructing user grouplevel records (aggregation).
These are then shared across channels/brands, to compute the desired estimate.
Our proposed estimation method is unbiased, but with a higher variance.
However by relaxing the need for constructing user-level data, it retains privacy, works without further assumptions and can achieve a wider coverage.
Our work borrows the idea of using multi-cluster experimentation from the work of Hudgens and Halloran [9].
Hudgens and Halloran [9] which proposed a two-stage randomization procedure to account for interference.
Our scenario however is different in that the same unit is receives multiple treatments instead of having exposure to treatments of other units.
Furthermore, we do not make exposure level assumptions.
PROBLEM STATEMENT
Let’s say that our two websites have users in set A and B .
Note that
A ∩ B ≠ ∅ ; so some users will visit both.
When third party cookies are available, these set of users will get a consistent view (in terms of experience/offers/ads etc); but without cookies this cannot be guaranteed, and a user may be exposed to multiple treatments.
For each user 𝑢 𝑖 ∈
A ∪ B we have an associated feature vector 𝑥 𝑖 .
Each user 𝑢 𝑖 ∈ A can be exposed to treatments 1 or 2 on website 1 .
Similarly, each user 𝑢 𝑖 ∈ B can be exposed to treatments 1 or 2 on website 2.
If a user 𝑢 𝑖 ∈ A ∩ B then they have exposure to treatments from both websites.
On the other hand those in A Δ B are only exposed to one website and we label their other exposure on the other website as “0”.
Since the two brands collaborate without sharing data, our approach takes the perspective of one of the two brands (since they can only analyze visitors on their website), without loss of generality, let’s consider the first brand.
For users in set A we have 6 potential outcome variables, i.e., 𝑌 1 , 0 ,𝑌 2 , 0 ,𝑌 1 , 1 ,𝑌 1 , 2 ,𝑌 2 , 1 ,𝑌 2 , 2 .
Of these variables only one is observed for each user based on the treatment they received and whether they visited both websites.
The problem is estimating the average treatment effect of shifting from treatment 2 to 1.
The challenges in estimation arises because of two main issues a) each website can only control allocation of treatments to their users; and b) the identity of the shared users is unknown.
Whenever a user 𝑢 𝑖 visits, the websites can only access the features 𝑥 𝑖 ; and choose a treatment without information on whether 𝑢 𝑖 ∈ A ∩ B or how the other website might allocate treatment.
Equivalently phrased we know for a user 𝑢 𝑖 of website 1 and allocated to treatmentthat we are observing one of 𝑌 1 ,. , 𝑌 1 , 1 or 𝑌 1 , 2 but not which of these.
On the other hand in standard causal inference we know for each observation unit, whether the observed outcome is 𝑌 1 or 𝑌 2 .
Let us make this more explicit: For a group which has been allocated treatment 1 by website 1, the expected average outcome is given by:
[Equation]
Here, ˜ 𝑌 1 is the random variable denoting the observed outcome of a visitor to website 1 who is randomized to treatment 1, 𝑝 is the fraction of users who are shared and hence visit both websites, while 𝛼 is the fraction of these shared users who receive treatment 1 on the second website as well.
Every user has the probability 1 − 𝑝 of only visiting website 1 and hence consistently receives treatment 1.
For these users the average outcome is E [ 𝑌 1 , 0 ] .
For the rest of the users (who are 𝑝 fraction of the population), an 𝛼 fraction of them are allocated to treatment 1 by website 2 (and hence are exposed to treatment pair 1 , 1).
The observed outcome on these users is E [ 𝑌 1 , 1 ] .
Similarly a 1 − 𝛼 fraction of the shared users receive treatment 2 from website and produce the average outcome
E [ 𝑌 1 , 2 ] .
The observed average effect is the probability weighted combination of all the contributions.
Furthermore by symmetry between the treatments, we can write a similar equation for the average outcome of group allocated treatment 2 by website 1.
The observed advantage of treatment 1 over treatment 2, i.e., E [ ˜ 𝑌 1 ]− E [ ˜ 𝑌 2 ] (which is also the standard treatment effect estimate) is given by:
[Equation]
Next, we analyze the treatment effect if one could track the users and provide them a consistent experience.
Every user has the probability 1 − 𝑝 of only being on website 1 and hence receiving consistently either treatment 1 or 2.
For these users the treatment effect is E [ 𝑌 1 , 0 − 𝑌 2 , 0 ] .
The rest of the users; who are 𝑝 fraction of the population; consistently receive treatment pair ( 1 , 1 ) or ( 2 , 2 ) .
The corresponding effect is E [ 𝑌 1 , 1 − 𝑌 2 , 2 ] .
The average treatment effect is the population weighted combination of the two contributions.
[Equation]
It is clear from the desired treatment effect and the observed treatment effect are mismatched due to contributions from the cross treatment outcomes 𝑌 2 , 1 and 𝑌 1 , 2 .
Moreover it is clear that the mismatch increases with 𝑝 the fraction of shared users.
In the next section we present our method of estimating the true treatment effect.
METHOD
We assume that we are able to make more than 1 macroclusters with a suitable mix of treatment allocation.
This is similar to the multi-stage randomization technique of Hudgens and Halloran [9].
Each user first gets assigned to a macrocluster which corresponds to one of the treatment allocation strategies.
Then within each such macrocluster randomization is done for allotment of treatment.
One possible way to solve this is to do stratified randomization with consistence across platforms/websites.
More specifically for each user
𝑢
𝑖 we put them in a category/macrocluster based on their features 𝑥 𝑖 , and this function or mapping the users to macroclusters is shared across all websites.
This can be achieved via something similar to FLoC or Topics API which allows advertisers partial view of the users preferences.
Note that privacy is still maintained here; since no website shares user-specific information with the other website.
Recall that the goal of this exercise is to estimate the following estimand of interest: ( 1 − 𝑝 ) E [ 𝑌 1 , 0 − 𝑌 2 , 0 ] + 𝑝 ( E [ 𝑌 1 , 3 − 𝑌 2 , 4 ]) .
This reflects the the following notation.
The treatments options at website 1 are 𝑇 1 and 𝑇 2 , and those at website 2 are 𝑇 3 and 𝑇 4 .
The notation 𝐸 [ 𝑌 1 , 0 ] denotes the expectation of the outcome in visitors who have seen 𝑇 1 at first website but have not visited website 2.
If we are in the most likely scenario where 𝑇 1 = 𝑇 3 and 𝑇 2 = 𝑇 4 , our estimand boils down to ( 1 − 𝑝 ) E [ 𝑌 1 , 0 − 𝑌 2 , 0 ] + 𝑝 ( E [ 𝑌 1 , 1 − 𝑌 2 , 2 ]) .
In the interest of keeping our solution generic, we are sticking to notation with four treatments 𝑇 1 , 𝑇 2 , 𝑇 3 , and 𝑇 4 .
Using this macro-level aggregation of user information, one can do randomization with the following approach.
Lets assume that we have only 2 macroclusters 𝐶 1 and 𝐶 2 .
In the first cluster the allocation ratio of the treatments T3 and T4 as chosen by the second website is 𝛼 ≠ 0 . 5; while in the second cluster the allocation ratio is 1 − 𝛼 .
We depict the observed average outcomes for cluster 𝐶𝑖 and treatment 𝑗 by the variable ¯ 𝑌 𝐶𝑖𝑗 (an estimate of E ( ˜ 𝑌 𝐶𝑖𝑗 ) ).
Then given our setting we have the four observed outcomes ¯ 𝑌 𝐶 1 1 , ¯ 𝑌 𝐶 1 2 , ¯ 𝑌 𝐶 2 1 , ¯ 𝑌 𝐶 2 2 .
Our estimator of the treatment effect is given by:
[Equation]
Claim 1.
ˆ TE is an unbiased estimate of the treatment effect TE
A proof of this claim along with and estimate of the variance of this estimator can be found in Appendix A.
4. Covariate Adjustment
Assuming the randomization is perfect, the earlier treatment effect estimate is unbiased unconditionally.
However when there is an imbalance in respect of some covariate between the groups, adjusting for the baseline effect leads to a more efficient estimator.
Next we provide a regression based method to improve the efficiency of our earlier estimator.
Algorithm
Input: Vector of outcomes 𝑌 𝐶 1 1 ,𝑌 𝐶 1 2 ,𝑌 𝐶 2 1 ,𝑌 𝐶 1allocation 𝛼
Output: Covariate adjusted treatment effect ˆ ATE cov (1) Let 𝑍 𝐶𝑗
𝑖 = 𝛿𝑌 𝐶𝑗 𝑖 i.e. it is 1 if 𝑌 corresponds to treatment 1and 0 otherwise (2) Fit OLS( [ 𝑌 𝐶 1 1 ,𝑌 𝐶 2 2 ] ∼ 𝑋 + 1 + [ 𝑍 𝐶 1 1 ,𝑍 𝐶 2 2 ]) (3) Let 𝛽 1 be coefficient of 𝑍 as estimated in previous step (4) Fit OLS( [ 𝑌 𝐶 1 2 ,𝑌 𝐶 2 1 ] ∼ 𝑋[-+ 
-]
+ 1 + [ 𝑍 𝐶 1 2 ,𝑍 𝐶 2 1 ] ) (5) Let 𝛽 2 be coefficient of 𝑍 as estimated in previous step (6) ˆ ATE cov
= 1 2 𝛼 − 1 ( 𝛼𝛽 1 + ( 𝛼 − 1 ) 𝛽 2 )
Our method is specified in Algorithm 1.
We fit two linear least square estimators; and combine the regression coefficients obtained from them.
We use the variable 𝑍 as an indicator of treatment allocation.
Unlike normal covariate adjustment where outcomes from the same group is chosen; here the model is fit from outcomes of different groups.
We fit model with outcomes of treatment 1 from cluster 𝐶 1 and outcomes of treatment 2 from cluster 𝐶 2 and vice versa.
The proof of correctness is presented in the Appendix B.
EXPERIMENTS & DEMONSTRATION
Description Consider the following scenario.
Website 1 and its affiliate website 2, want to do a synchronous change (for example put the same banner on both websites).
Ideally with third party cookies this experimented can be conducted in a standard manner.
However without such identifier both websites have to individually randomize their treatments.
5. Synthetic Data
We first conduct simulation experiments where, by design, all parameters are known and adjustable.
We can then quantitatively measure the performance of our method across different ranges of available parameters.
This simulation was conducted by generating 10000 observations, and each experimented was repeated 20 times.
of 20 different
Poisson variables.
Each individual potential outcome was obtained via a noise-corrupted Gaussian distribution.
Furthermore each outcome is also influenced by a user specific covariate.
Specifically, each potential outcome variable 𝑌 𝑖𝑗 at a single observation unit is obtained via a linear model from covariates 𝑋
The true treatment effect in this case is ( 1 − 𝑝 )[ 𝜇 1 , 0 − 𝜇 2 , 0 ] + 𝑝 ( 𝜇 1 , 3 − 𝜇 2 , 4 ) .
We vary two parameters 𝛿 1 = 𝜇 1 , 4 − 𝜇 1 , 3 and 𝛿 2 = 𝜇 2 , 4 − 𝜇 2 ,
Results We conducted these simulations and measured the error in the estimated ATE versus the true ATE for four methods.
These include the standard ATE estimate (uncorrected) , standard ATE with covariate adjustment (uncorrected + adj), our ATE estimate (corrected) and out ATE estimate with covariate adjustment (uncorrected+adj).
Due to inherent variability caused by sampling, there will always remain some variability in the estimate.
We present in
Figure 2 the results from the experiments.
The plots depicts both the bias of the estimator and the standard error of the estimator.
It is clear that the corrected method provides unbiased estimator of the true ATE while the uncorrected estimator is biased.
Furthermore as expected our proposed covariate adjustment method gives an unbiased estimate.
[Figure]
[Equation]
[Figure]
[Figure]
5. Observational Data
Next we conduct experiments from the experimental logs from
Target, an experience cloud product for designing customers’ experience.
Each entry in the data corresponds to a user visit, and records the users conversion event (which is our targeted binary response) along with the specific experience which the user was served and a variety of covariates such as session duration, browser details, time etc.
There are six possible experiences which were regrouped for our purposes into two treatments.
Scenario
We split the total time period into two halves, and considered the visitors in the two time periods as visiting two separate websites.
This allowed us to simulate the visits on two websites with related treatments.
A user who is present in both time periods is a user with exposure to both parts of the treatment, while a user who is present in only one time period corresponds to a user who visits only one website.
Splitting the records in a such a way we can create a joint distribution of outcomes and treatments from which samples can be obtained.
Scenario
We also ran another experiment where we directly isolated users who got exposure directly to multiple treatments and tried estimated ATE from these experiments as well.
Since the outcomes in this case are sampled from historical logs, the effect of interference is fixed and cannot be changed.
However one can analyze the effect of changing the fraction of users who receive exposure to multiple treatments.
Results We conducted these simulations and measured the error in the estimated ATE versus the true ATE.
We present in Figurethe results from the experiments.
The plots depicts both the bias of the estimator and the standard error of the estimator.
The actual data has less than 0.1% users who received exposure to different treatments, which means the bias in the historical record is minimal.
As such we present the results of a simulation done by resampling the records.
[Figure]
As expected the bias of the standard estimator increases as the probability of user having multiple visits increases.
This is in line with the theoretical analysis earlier, which shows that the error is proportional to 𝑝 ( the probability of interfered outcome).
5. Demonstration Plan
We develop a demonstration of our technology that shows how a marketer for a brand collaborating on conducting an A/B test jointly with another brand can design and analyse an experiment by only sharing the parameters of the experiment.
In Figure 4 the marketer for Brand 𝐴 specifies the allocation ration of 𝛼 .
The chart shows the estimated treatment effect estimated from the naive
Our approach suggests that there is no treatment effect (which is the truth in the simulated scenario), while the naive estimate would suggest a negative treatment effect.
Similarly, we show the experience for brand 𝐵 in Figure 5.
CONCLUSION
We have proposed a two-stage experimental study design that two brands can use to jointly test the treatment effects from an experiment.
Our approach has no dependence on third-party cookies, and does not require any individual level information to be shared between the two brands.
We show that the proposed ATE estimate is unbiased and give a theoretical formula for the variance of our estimate.
We additionally show that we can compute the regression adjusted estimate that can provide additional information on the
ATE.
Additional work is needed to address multiple treatment arms to control for multiple hypothesis testing.
It will also be interesting to explore the effect of identity fragmentation on a single brand’s website.
A BASIC ESTIMATOR
Our estimator of the treatment effect is given by:
[Equation]
Claim 1.
ˆ TE is an unbiased estimate of the treatment effect TE
Proof.
We can see from the earlier discussion that the expected values of the average observed outcomes is given by:
[Equation]
Now by linearity of expectations:
[Equation]
Plugging in the expectations from the earlier equations gives:
[Equation]
which is the desired treatment effect .
□
The average outcome in each is a combination of the potential outcomes weighted by their relative weights.
We have three outcome values, and three effects to estimate: which means that generically these can be uniquely solved.
A. Uncertainty Analysis
Since the estimation process is just a linear combination of different average quantities, we can provide an easy upper bound to the variance of our estimator in terms of variance of outcomes.
Let 𝑉 𝑀 and 𝑉 𝑚 be the maximum and minimum variance among all outcomes i.e 𝑉 𝑀 = maxVar ( 𝑌 1 , 0 ,𝑌 2 , 0 ,𝑌 1 , 3 ,𝑌 1 , 4 ,𝑌 2 , 3 ,𝑌 2 , 4 ) and 𝑉 𝑚 =
minVar ( 𝑌 1 , 0 ,𝑌 2 , 0 ,𝑌 1 , 3 ,𝑌 1 , 4 ,𝑌 2 , 3 ,𝑌 2 , 4 ) .
Then, 𝑉 𝑚 √ 𝑛 ≤ 𝑉𝑎𝑟 ( ¯ 𝑌 𝐶𝑖𝑗 ) ≤ 𝑉 𝑀 √ 𝑛 where 𝑛 is the number of observations in each group.
and
[Equation]
From above expression it is clear that the minimum variance is obtained at 𝛼 = 0 , 1.
This is not too surprising since that corresponds to an
AA vs AB test across the two websites.
In such a case we can with certainity find a set of users who exactly receive one treatment (i.e. either 1,3 or 2,4), and the results across the two clusters can be combined to obtain the treatment effect.
An issue however for bounds based on the above variance terms is their validity, since the estimates are only asymptotically normal.
However we also note that each average observed effect is an average of outcomes from different users.
Hence under independence of each user, Chernoff-Hoeffding bound can be used to provide non-asymptotic intervals for each experiments.
Since our final estimator is a linear combination of different average outcome effects, we can trivially obtain valid confidence intervals from the Hoeffding bounds of each individual experiment.
A. Estimating conditional effects
Often businesses would be interested in conditional effects as they want to focus more on users who have potentially high conversion rates.
From the expression , it is obvious how to obtain conditional average treatment effect (CATE) from the data.
As long as conditional outcomes for all four groups can be obtained the earlier expression can be used to obtain the conditional effect by replacing average outcome by conditional average outcomes.
These conditionals can be estimated via non-parametric regression, matching or via propensity weighted estimators.
[Equation]
Claim 2.
ˆ CATE is an unbiased estimate of the conditional average treatment effect CATE
Proof.
The proof of the above statement is analogous to our earlier proof.
□
B COVARIATE ADJUSTMENT
Let the numbers of subjects randomized to experimental treatment and control be 𝑛 1 = (cid:205)
𝑛𝑖 = 1 𝑍 𝑖 and 𝑛 1 = (cid:205)
𝑛𝑖 =
1 ( 1 − 𝑍 𝑖 ) , 𝑛
= 𝑛 0 + 𝑛 1 .
The sample means of outcome in each group is then given by:
[Equation]
Then direct usage of the the OLS formula shows that shows that the least squares estimator for 𝜂 𝑍 is
[Equation]
𝑥𝑌
[Equation]
𝑥𝑋 𝑋
[Equation]
(4) where 𝑑 1 = 𝑛 (cid:213) 𝑖 =(cid:0) 𝑍 𝑖 − ¯ 𝑍 (cid:1) 𝑋 𝑖 , (cid:99)(cid:213) 𝑥𝑦 = 𝑛
[Equation]
𝑛 (cid:213) 𝑖 =(cid:0)
𝑋 𝑖 − ¯ 𝑋 (cid:1) (cid:0) 𝑌
[Equation]
− ¯ 𝑌 (cid:1) , and (cid:99)(cid:213)
[Equation]
𝑇
Now by Slutsky’s theroem, we know that ˆ Σ 𝑋𝑌 and ˆ Σ 𝑋𝑋 converge in probability to their expectation counterparts ( i.e. the cross-covariance and covariance matrices respectively).
Similarly, 𝑛 2 /( 𝑛 0 𝑛 1 ) 𝑝
→ { 𝛿 ( 1 − 𝛿 )} − 1 , and 𝑛 − 1 𝑑 1 𝑝 → 0.
Hence the first term in (4) asymptotes to 1; while the second term becomes
[Equation]
Now since 𝑍 is independent of 𝑋 , the expected value of the second term in the above equation is
[Equation]
Thus asymptotically the coefficient of 𝑍 connverges to the standard estimator.
Leon et al.
[13] details conditions under which the above estimate has a lower variance.
Furthermore well known result in regression theory [7] imply that the above estimator is consistent and asymptotically normal under entirely unrestrictive conditions.
C ASSUMPTIONS
Assumption 1 (Strong ignorability).
Let 𝑍 𝑗𝑖 be a random variable denoting the treatment allocated to user 𝑖 on site 𝑗 .
[Equation]
This assumption first explicitly introduced by Rosenbaum and Rubin [21] is standard assumption for causal effect estimation under the potential outcomes approach.
This assumption makes the covariates 𝑋 admissible or deconfounding.
Under strong ignorability, treatment effects can be estimated without bias using propensity weighting as shown in Pearl [20].
Assumption 2 (Positivity).
< 𝑃 ( Z 𝑗𝑖 = 𝑡 ) < 1 for all users 𝑖 , sites 𝑗 and treatments 𝑡 .
This requirement of positivity (also known as overlapping) ensures that every treatment allocation is possible, or equivalently
each unit 𝑖 has a non-zero probability of being allocated any treatment.
Since the different channels are going to independently allocate treatments to any user, this can be easily ensured that there is some randomization at each website.
Assumption 3 (Independent Choice).
If 𝐴 𝑖 is a random variable denoting the choice of a common user to visit both websites, then 𝐴 𝑖 ⊥ 𝑌 𝑗𝑖 ( 𝑡 ) ∀ 𝑡 .
While this is implicitly subsumed under the strong ignorability assumption, we make this explicit.
The reason for this is that strong ignorability is often used instead of no unobserved confounder.
In our scenario no website actually has information about 𝐴 𝑖 , as they do not know if an individual user will visit the other website, and hence 𝐴 is unobserved.
D ESTIMATING TREATMENT EFFECT ON ONE CHANNEL
[Figure]