Paper: A/B testing under Identity Fragmentation (ICLR-2024)

Reviewer comment (Experiments):
Although the authors address the potential issues that could arise from inaccurate neighborhoods, it's mainly focused on the removal/addition of edges. I am curious about a setting where the edge exists but is weak. For example, assume a user owns several devices (e.g., same login credentials for Netflix), however, in principle, there are several *different* users using the same credentials. Would it be better to use a probabilistic matrix and rely on a threshold to decide the neighborhood?

Author response:
Would it be better to use a probabilistic matrix and rely on a threshold to decide the neighborhood?
* We draw the reviewers attention to Remark 1 in the paper where we mentioned something along this line of using confidence in an edge to determine neighbourhoods. Conservatively choosing a threshold for inclusion of an edge, will likely result in including a number of extraneous edges, which motivated assumption A7. 
The problem comes from missing an influence which can lead to biased estimates. From Figure 5.3 we can see that there is a potential bias-variance tradeoff with including few edges increasing bias while adding more edges adding variance. One can try to choose the threshold for including an edge based on the importance of these two factors.

Reviewer comment (Theory):
While the authors address the issue of varied treatments on devices in the neighborhood, I am curious about a case where the actions determining Y are completely different on each device in N (e.g., in the case of shared Netflix passwords accessed by different members.). How would that affect Y?

Author response:
the actions determining Y are completely different on each device in N ?
* Assumption A2,A3 support heterogeneity i.e. behaviour on the nodes belonging to the same user need not be similar. Specifically, even for the same treatment allocation z and the same neighbourhoods, two devices can have different true exposures (as $\phi$ depends on $X_i$). However, since our method does not compute true exposures, it cannot determine which nodes are actually influencing the outcomes.

Reviewer comment (Experiments):
In practical settings, even though users might own different devices, in most cases, they are logged/active on one at a time or restricted to a single access at a time. Would adding the temporal (time) aspect of the model improve treatment estimates?

Author response:
Would adding the temporal (time) aspect of the model improve treatment estimates?
* This is an interesting question, which is a potential future direction of research. While time-series will provide more information, lack of independent errors mean that the result in this work does not ensure identification of the model.

Reviewer comment (Writing):
There are some write-up issues and typos. Although this can be implied, some variables are not immediately defined when used, e.g., in A2. On page 8, the last sentence before section 5.2 seems incomplete. On page 6, it should |M(i)| >> |N(i)| not |M(i)| >> N(i), page 4 last paragraph, "treatment group in small" should "treatment group is small", e.t.c.

Author response:
Typos and Conclusion
* We have fixed the typo issues and slightly reformatted the paper to address some other points raised by reviewers (such as adding conclusion)

Reviewer comment (Presentation):
Some of the crucial sections are missing. I think having a conclusion/discussion and limitations would help address potential issues that might arise, for example, from unsatisfied assumptions and broader recommendations. I understand this might be a space issue, but if authors can find a way to add them, that might be helpful.

Author response:
Typos and Conclusion
* We have fixed the typo issues and slightly reformatted the paper to address some other points raised by reviewers (such as adding conclusion)

Reviewer comment (Theory):
Other issues, mostly minor: Authors could improve the font of the figures. What do the authors mean by type in this sentence "if the listing and person have the same type" on page 8? In the age of GDPR and similar privacy measures, device linking might not just fall short but might be an altogether infeasible approach. Lastly, I find the explanation given for the strong assumption (A7) a bit unrealistic.

Author response:
Other issues
* We have added information about this in the Experimental Section, with greater details added in the Appendix. We would also like to draw your attention to the fact that the purpose of this paper is precisely to estimate treatment effects without resorting to identity linking. The superset assumption is motivated exactly by the failure to do identity linking. Since, coarse information can be used under certain contexts, one can create these supersets with such coarse information. Our method maintains privacy, while allowing for estimation of GATE.

Reviewer comment (Novelty):
the novel of the theoretical results should be verified. I am not sure which thm is essential and novel in the work. I am not sure whether the prop. is directly from given related works.

Author response:
*  The novelty of this work is in demonstrating identifiability of GATE under identity fragmentation which has not been done before. The second novelty comes in considering the case of imperfect knowledge of the device graph which has also not been done. We do so via transforming this problem into a noisy regression problem. The proposition is a novel claim of our work. Its derivation relies on a theorem (stated in Theorem1) from earlier work [4].

Reviewer comment (Experiments):
more experiments results need to be provided with larger dataset and the more complex real world settings.

Author response:
* Since the use case is motivated by application in online experimentation, data for these settings is usually private or unavailable. As such we have to rely on simulations and semi-synthetic data. Furthermore, unlike standard problems like regression, the underlying ground truth is unavailable, necessitating use of simulated data to be able to verify the predictions. This is also the approach used in related works [5]. We would also like to draw your attention to the AirbNb simulation has been generally used for a variety of papers dealing with experimentation on digital marketplace [2,3].
However, to incorporate the reviewers suggestion, we have evaluated our method on a real observational air-pollution dataset used in [1]. Its setting is slightly different from ours, as it focuses on spatial interference from neighbouring power plants. But we can adapt our method to the use case. We have added these results in the Sec 5.4, and added details in the supplementary material.

Reviewer comment (Theory):
The analysis in the work is all about the linear model, how about non-linear ones?

Author response:
* Q1) Our model is linear in computation of exposure but non-linear in exposure itself. We draw your attention to the function $g$ in A2 which is a potentially non-linear function applied to the exposure values.

Reviewer comment (Novelty):
how are the proposed methods different from the classical VAE, if so, if more advanced ones can be adopted?

Author response:
* Q2) Our contribution is in highlighting how identity fragmentation invalidates standard estimators used in A/B testing, and how one can still identify the treatment effects by suitable transformation of the problem. This is important as without identifiability, learning model parameters need not be valid in deriving treatment effects. Next we used the VAE-based method to estimate the treatment effect. In principle other methods for variational inference like normalizing flows etc can also be used for learning the model parameters.

Reviewer comment (Theory):
will the noise distribution influences the methods; for the time series data, will the noise correlation or process take effects?

Author response:
* Q3) The noise distribution as long as independent should be tractable in our setting. While in principle one can posit a model for time-series outcomes as well, it is not clear that identifiability of the mean function holds with non-independent errors. Analysing that is an important direction of future research.

Reviewer comment (Writing):
Some variables, such as X, lack clear definition or explanation, particularly in their role in estimating true neighbors within the model. This lack of detail or oversimplification can lead to confusion when attempting to understand the motivation and core concepts presented in the paper, significantly impacting its quality.

Author response:
* Regarding the relationship between the covariate X and the randomly generated "random device graphs," X plays no role in determining neighbourhoods $M(i)$. We see that it might not have been sufficiently clear in the manuscript, but our approach is agnostic to how $M(i)$ is obtained.
In our experiments,X pertains to device-level covariates influencing outcomes conditional on treatments, such as location, phone type, demographic information, or other relevant details available to marketers at the device level. Although X could theoretically be used to determine neighborhoods M(i), our approach remains agnostic to the method of obtaining M, treating it as a given parameter. We have explained this better on Page4 of the revised version.

Reviewer comment (Reproducibility):
The utilization of the trained model, specifically in the format of equation A2, within the estimation process of the GATE value is not elucidated. This omission introduces ambiguity when attempting to compare it with previous methods in the experimental section.

Author response:
* $\tau$ is defined via Equation 1. One the variational model p_\theta is estimated, $Y_i$ is directly specified by p_\theta(Y_i|X_i,z). We draw your attention to Sec 4.2, where the exact probabilistic model is mentioned. Since p_theta directly specifies the means $\mu_Y$, 
One then estimates $\tau $ as  $1/n \sum \mu(Y|\vec{1},X_i) - \mu(Y|\vec{0},X_i)$, which is just equation 1, where the potential outcomes are replaced by the estimate $\mu$ obtained from p_\theta.
We have now added these details in Sec 4.2 and provided more information in the supplementary material.

Reviewer comment (Experiments):
The settings appear to intentionally align with the assumptions of the proposed model without practical justification.

Author response:
* The experiments in Sec 5.1, are to demonstrate the correctness of our method when all its assumptions hold. The similarity between the generative model in 5.1 and A2, is so that the probabilistic model is correctly specified. We would also  like to highlight that A2 and A3, are general enough to cover many sort of exposure models used in literature [1,2,3]. We have made this clear on Page 5.
Furthermore, in Section 5.2, we experiment with the airbnb model which does not satisfy A2. (In fact it does not satisfy any exposure assumption). In such a case one can get biased estimates, but as can be seen from Figure 4, all other estimates (except HT) are also biased, and HT requires knowledge of the true graph.

Reviewer comment (Reproducibility):
The experimental environments are not clearly outlined, including specific settings of the baseline estimators and the methodology used for calculating metrics in the figures.

Author response:
*  DM and Poly-regression in our experiments stems from their common usage as estimators in the literature. Poly-regression fits a polynomial between exposure (typically the number or fraction of treated neighbors) and observed outcomes, while DM represents the classic difference in mean estimate discussed in Section 3 under SUTVA. We have incorporated this information into the supplementary material, providing additional experimental details for clarity.

Reviewer comment (Reproducibility):
In equation A2, c0, c1, g, and w are referred to as neural network functions, yet no further details are provided. This omission makes it challenging to infer the specific characteristics or behavior of the proposed approach.

Author response:
1 We used single layer MLPs for all the functions. We have added this in the revision (Remark 3)

Reviewer comment (Writing):
The relationship between the covariate X and the randomly generated "random device graphs" remains unclear. Given the methodology described, it appears that X plays a vital role in determining neighboring relationships. A more detailed explanation is needed to elucidate this connection.

Author response:
* Regarding the relationship between the covariate X and the randomly generated "random device graphs," X plays no role in determining neighbourhoods $M(i)$. We see that it might not have been sufficiently clear in the manuscript, but our approach is agnostic to how $M(i)$ is obtained.
In our experiments,X pertains to device-level covariates influencing outcomes conditional on treatments, such as location, phone type, demographic information, or other relevant details available to marketers at the device level. Although X could theoretically be used to determine neighborhoods M(i), our approach remains agnostic to the method of obtaining M, treating it as a given parameter. We have explained this better on Page4 of the revised version.

2a X plays no role in device graph generation. We independently create a device graph and populate covariates X at the nodes. While in principle one can use the node level X to guess the device graph, we simply assume the existence of the device graph. This is because, as long as A1-7 hold, the model remains identifiable, irrespective of how the estimated device graph $M$ was created. That said practically, having poorer/dense graphs adds significant variance to the estimates. Since we assume $M$ to be given in all our experiments, covariates X are simply used as additional covariates in specifying the mean function parameterizing the outcomes.

Reviewer comment (Experiments):
The similarity between the equation for generating ground truth and model A2 raises questions. Is there a specific reason for this resemblance, and could it potentially confer an advantage to the proposed model in certain settings? Further clarification on this matter would be beneficial.

Author response:
* The experiments in Sec 5.1, are to demonstrate the correctness of our method when all its assumptions hold. The similarity between the generative model in 5.1 and A2, is so that the probabilistic model is correctly specified. We would also  like to highlight that A2 and A3, are general enough to cover many sort of exposure models used in literature [1,2,3]. We have made this clear on Page 5.
Furthermore, in Section 5.2, we experiment with the airbnb model which does not satisfy A2. (In fact it does not satisfy any exposure assumption). In such a case one can get biased estimates, but as can be seen from Figure 4, all other estimates (except HT) are also biased, and HT requires knowledge of the true graph.