Paper: Independent-Set Design of Experiments for Estimating Treatment and Spillover Effects under Network Interference (ICLR-2024)

Reviewer comment (Theory):
This paper attempts to design a more robust methodology towards this idea but relies on knowledge of the underlying network G. In practice this is never known and the paper does not consider the case of a misspecified G.

Author response:
In our paper, we assume the network is observed at least before the partitioning step. This is usually the case. For example, the interference can the friendship network in social networks or the transportation network for the supply chain problems. It is a common assumption (see Ugander et al., 2013; Forastiere et at., 2021; Cai et al., 2022)  Our work focuses on observed network, and the design of experiment is on observed networks, but it is a good direction for the future work when network G is misspecified,  and we include that in our discussion as future work.

Reviewer comment (Theory):
How does this method work when G is misspecified?

Author response:
In our paper, we assume the network is observed at least before the partitioning step. This is usually the case. For example, the interference can the friendship network in social networks or the transportation network for the supply chain problems. It is a common assumption (see Ugander et al., 2013; Forastiere et at., 2021; Cai et al., 2022)  Our work focuses on observed network, and the design of experiment is on observed networks, but it is a good direction for the future work when network G is misspecified,  and we include that in our discussion as future work.

Reviewer comment (Experiments):
It would be good to see how this could be applied in a practical example with different underlying graphs.

Author response:
We agree it is always helpful to inspect models on real data. However, the paper focus more on the design and theory— while in real data we never know the true potential outcomes and true effects. Therefore, we compare our design with other designs on the synthetic datasets, where we can artificially set the truth and compare the performances.

Reviewer comment (Theory):
How does optimizing over treatments affect the randomization assumption? Could this affect the internal validity of the study if randomization is weakened (i.e., in the vein of Bugni, Canay, and Shaikh 2017)? Moreover, could there be distributional differences between the independent set and auxiliary sets?

Author response:
If we understand the reviewer’s question correctly, it is about the representativeness of the outcomes after a deterministic assignment of the treatments. We admit that, for a given sample set (that is a fixed potential outcomes and a fixed observed network), the greedy algorithm and the optimization of assignments do indeed weaken the representativeness of the observed outcomes because the vertices has unequal probabilities of being assigned to treated or being partitioned to the independent set. This is why we are NOT claiming our estimation is unbiased for the given sample set of data. Instead, we DO claim the unbiasedness from the super-population perspective (Assumption 2), where all samples along with their potential outcomes and their network are assumed to be sampled from the population. We would refer to Section 7.3 of Imbens and Rubin (2015) for the super-population framework. This is related to our response to point 9 of reviewer DrFU.

Reviewer comment (Presentation):
Inconsistency: Why does Figure 2 use n=60 whereas Table 2 uses n={100,200,400}?

Author response:
Figure 2 illustrates a detailed visualization of confidence intervals to show how bias and variance for different designs change when interference gamma increases. We use n = 60 as an illustration example. Table 2 provides more comparisons of different designs on distinct random graphs when interference is fixed, hence we let n = {100, 200, 400} to introduce diversity.

Reviewer comment (Presentation):
Why is Graph Cluster omitted from Figure 2?

Author response:
We add the simulation results of  graph cluster randomization to the revised manuscript in Figure 2.

Reviewer comment (Reproducibility):
Graph Cluster is not mentioned in the Section 5 introduction; what cluster design is used?

Author response:
Graph clustering is same as the randomized saturation design except that the user needs to cluster the units when no clear cut-offs exist. We use multi-level modularity optimization(Blondel, Vincent D., et al, 2008) for the clustering. Since there is no clear cluster boundary in the network, graph clustering is inferior to the independent set design due to significant bias.To better illustrate the performance of graph clustering,  we add the simulation results of graph cluster randomization to the revised manuscript.

Reviewer comment (Experiments):
Is there a reason some designs mentioned in Section 2 (ego-clusters, randomized saturation) appear to be omitted?

Author response:
Some of the designs listed in Table 1 are widely used designs, which were not necessarily designed for the causal inference over a well-connected interference network. The performance of other designs in Table 1 are obviously inferior to our approach. That is why only a few methods are compared for demonstration purpose. Additional results of comparison with other designs in Table 1 have included in the revised manuscript, include ego-clusters and graph cluster (similar to randomized saturation).

Reviewer comment (Experiments):
Given the discussion in Section 4.2, how do results vary for more varied p/s specifications for ER random graphs?

Author response:
Thm 1 from Section 4.2 provides a lower bound for the size of the independent set, which suffices to show its superiority over other methods. In the simulation part, the variance roughly scales as 1/n, which is much better than the lower bound provided in Section 4.2.

Reviewer comment (Experiments):
How robust are the results to the size of the independent set?

Author response:
The variance scales as 1/n_I from the theoretical perspective.

Reviewer comment (Reproducibility):
What is the $\rho$ chosen for the IS design in Section 5.2? Is it the usual IS setting from Karwa and Airoldi where $\rho = 0$?

Author response:
in 5.2 we let \rho = 0 when estimating  ATE(the average direct effects)

Reviewer comment (Writing):
Minor nomenclature question: The direct effect should be $\tau_{i}^{(d)}(0)$ and $\tau_{i}^{(d)}(\rho)$ represents the total effect in a partial spillover situation, correct?

Author response:
For the nomenclature question, yes, they can be considered as the direct effect and the total effect.

Reviewer comment (Theory):
How reasonable is Assumption 2 given the greedy algorithm to construct $V_I$ has no concern for representativeness?

Author response:
In practice, any sub-sample/separation design involves the representativeness issue on a fixed sample dataset. On the one hand, the first step in the greedy algorithm involves a random choice of the first vertex. Every vertex has a positive probability of being selected to the independent set. The bias from the greedy algorithm is alleviated by the stochastic algorithm, similar to previous work, such as Saint-Jacqueset al., 2019;  Uganderetal.,2013.  On the other hand, we are aiming to estimate the population causal effect instead of the sample averaged causal effect. A bias in the representativeness of the independent set to the sample set is mitigated by considering the super-population perspective, where, under repeated experiments, the network and the potential outcome are randomly drawn from the population. Therefore, the estimator is unbiased for the populational causal effect in the super-population framework (instead of the unbiasedness for sample average effect).

Reviewer comment (Novelty):
The main weakness in this paper is the lack of a clear comparison to related works both theoretically and numerically. For example, how does the new theoretical guarantees improve over previous works? Otherwise, the analysis looks like an application of linear regression estimator.

Author response:
We listed a few other designs in the manuscripts for comparison, which includes the completely randomized design (CRD), randomized saturation design/graph clustering design (RSD), ego-clusters design (ECD). Neither of CRD or RSD is suitable for the networks described in the manuscript, which are well, though sparsely, connected. The reasons are (1) in CRD, most of the vertices receive an interference level as 1/2, resulting in a huge variance in the regression estimator;  (2) RSD requires isolated clusters — otherwise, significant bias arises. Only the ECD aims to solve a similar problem. However, as we illustrated in Table 1, at least for Erdos-Renyi random graphs, ECD is expected to have a smaller sample size compared to our methods. In summary, compared to CRD and RSD, which were developed for other problems, our method guarantees a smaller bias and smaller variance. Compared to ECD, our method achieves a larger effective sample size, resulting in a smaller variance (see e.g. Thm 3).

Reviewer comment (Theory):
One contribution the paper claimed is using fewer assumptions for this model, it would be better to describe this more clearly. For example, what assumptions can be removed compared to previous works?

Author response:
As we mentioned in the above response, compared to the well-known CRD, we do not require SUTVA, that is, we allow for interference. Compared to RSD (which is widely used in networks with interference), we relax the partial interference assumption to a heterogenous version in Assumption 1, and we do not require an isolated clustering structure of the network for the estimator to be unbiased.

Reviewer comment (Theory):
The results rely on the greedy algorithm 1 to have a decent performance. Theorem 1 only gives the lower bound on ER graph which seems to limit the application of this framework.

Author response:
We admit that the current lower bound is developed for Erdos-Renyi random graph. The distribution of the sampled network is determined by both the populational network and the sampling procedure, which is way too complicated to be discussed in this manuscript. Therefore, we establish the current partition result (Thm 1) on Erdos-Renyi network, while other theorems remain valid for other networks. Performance on other common random networks are evaluated through simulation instead.

Reviewer comment (Experiments):
The simulation results are comparing only to completely randomized design. Is it possible to compare with other designs cited in the introduction and Table 1?

Author response:
Ego-clusters didn’t optimize the selection of egos and can be only applied to estimating spillover effect. To further illustrate the performance of Ego-clusters, we have included new simulation results of Ego-clusters design to the revised manuscript pdf in the updated Table 2 and Figure 2.

Reviewer comment (Writing):
There are some typos, for example, in section 4.3 and 4.3.2, it is referring to section 2.3 and 2.3.2 which does not exist. (Should be 3.3 and 3.3.2?)

Author response:
Thanks for pointing out the inconsistency of the section numbers, we modified the section numbers.