Paper: Gene Regulatory Network Inference in the Presence of Dropouts: a Causal View (ICLR-2024)

Reviewer comment (Theory):
Most of the theoretical analysis is deferred to the appendix, which can sometimes lead to questions due to insufficient detail.

Author response:
-   **Theoretical analysis deferred to appendix:**  We are keeping improving the way to deliver our results comprehensively. Our current structuring was due to page limit and the relatively dense contents in this paper -- we chose to focus the main body on key messages, illustrative examples, and experimental highlights. Many theoretical details, though very interesting (e.g., testability of A3), had thus to be deferred to appendix.

Reviewer comment (Presentation):
The references are a bit sloppy.

Author response:
- **Formatting errors in references:**  We thank you for pointing out these errors. We have thoroughly reviewed the references and corrected the formatting issues as much as possible in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h).

Reviewer comment (Theory):
Page 4, Example 4: The Bernoulli distributions should be reversed, since there should be a minus sign in the denominator exponential when computing the logistic function.

Author response:
- **Bernoulli distribution in Example 4:** We appreciate your careful examination. Please kindly note that upon double check, we believe the current one is indeed correct: the term in Bernoulli(·), e.g., logistic$(\log 2 - Z_i)$, is the probability of dropout ($D_i = 1$), which is why the density function of $Z_i$ is multiplied not by this term, but by one minus it, i.e., logistic$(-(\log 2 - Z_i))$. We have replaced the name "logistic" with "sigmoid" in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h) for clarity.

Reviewer comment (Writing):
Page 10, reference typo: * after "Tabula Sapiens Consortium".

Author response:
- **Formatting errors in references:**  We thank you for pointing out these errors. We have thoroughly reviewed the references and corrected the formatting issues as much as possible in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h).

Reviewer comment (Writing):
Page 11, formatting error: "ALBERTS" should not be capitalized.

Author response:
- **Formatting errors in references:**  We thank you for pointing out these errors. We have thoroughly reviewed the references and corrected the formatting issues as much as possible in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h).

Reviewer comment (Novelty):
Page 11, there seems to be no reference to Gao et al. (2022) in the paper.

Author response:
- **Formatting errors in references:**  We thank you for pointing out these errors. We have thoroughly reviewed the references and corrected the formatting issues as much as possible in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h).

Reviewer comment (Presentation):
Page 12, duplicate author: "Paul R. Rosenbaum".

Author response:
- **Formatting errors in references:**  We thank you for pointing out these errors. We have thoroughly reviewed the references and corrected the formatting issues as much as possible in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h).

Reviewer comment (Theory):
On page 6, when discussing the connection between BIC score and Fisher-Z test statistics, how is it concluded that "only local consistency is actually needed"? The reference in Nandy et al. (2018) shows this connection, but does not say anything about the assumptions made in the GES paper. Clarification on this point is requested.

Author response:
**(Q1)** The reviewer requests clarification on the claim that "(for the correctness of GES), only local consistency of the scoring function is actually needed".

**R:** Thank you for your feedback on this point. We have revised this part in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h) to make the logical connection clearer. Here is a summary: first, the fact that only local consistency is required for GES's consistency can already be seen from Chickering (2002)'s proof. Specifically, their Lemma 9 and Theorem 4 (main results) are grounded on local edge modifications. Then, we mention Nandy et al. (2018) as they make this point more explicit: they derive the BIC score change directly as the partial correlation involved in local consistency definition (Lemma 5.1), and further go beyond BIC score, suggesting "defining generalized scoring criterions by... conditional independence". This is recognized and implemented by [4] at the same time, and later by [5], [6].

Reviewer comment (Theory):
On page 7, clarification is needed on the meaning of "identifiability upper bound" for identifying the dropout mechanisms and how this upper bound is characterized.

Author response:
**(Q2)** The reviewer wonders for some technical details regarding the identifiability result without assumption A3 (i.e., when dropout can be affected by other genes). Specifically,

> "What is the 'identifiability upper bound' for identifying the dropout mechanisms?"

**R:** The identifiability upper bound of dropout mechanisms is characterized by Proposition 2 in Appendix C.1: "If $Z_i$ and $Z_j$ are truly adjacent in GRN, then whether they affect each other's dropout, i.e., whether edges $Z_i \rightarrow R_j$ and $Z_j \rightarrow R_i$ exist, is naturally unidentifiable". This is because in this case, the (in)dependence between e.g., $Z_j$ and $R_i$ given at least $Z_i$, is untestable (see Proposition 3, Testability of CI queries).

Then, what if $Z_i$ and $Z_j$ are truly non-adjacent in GRN? Theorem 2 indicates that this non-adjacency, together with the absence of $Z_i \rightarrow R_j$ and $Z_j \rightarrow R_i$, can be identified barring one particular case. So,

> "What is this 'one particular case'"?

**R:** This particular case is when the non-adjacent $Z_i$ and $Z_j$ **always share a common child** (not descendant -- so we say it as so *particular*) that has to be conditioned on. This is formally defined in Proof of Theorem 2: "For every $\mathbf{S} \subset \mathbf{Z}$ that d-separates $Z_i$ from $Z_j$, there is an open path $Z_i\rightarrow W \leftarrow Z_j$, where either $W \in \mathbf{R}_{\mathbf{S}\cup \{i,j\}}$, or there exists a $k \in \mathbf{S}\cup \{i,j\}$ s.t. $W$ is $R_k$'s ancestor". Only in this particular case, the true d-separation cannot be recovered, resulting in a false adjacency.

We thank you for your detailed interest in these aspects of our work. For further clarity, we hope you will find Appendix C, particularly the illustrative examples (Figures 8 and 9), helpful.

Reviewer comment (Theory):
In Theorem 2, clarification is needed on what is the "one particular case" in which Z_i and Z_j are non-adjacent in the underlying GRN.

Author response:
**(Q2)** The reviewer wonders for some technical details regarding the identifiability result without assumption A3 (i.e., when dropout can be affected by other genes). Specifically,

> "What is the 'identifiability upper bound' for identifying the dropout mechanisms?"

**R:** The identifiability upper bound of dropout mechanisms is characterized by Proposition 2 in Appendix C.1: "If $Z_i$ and $Z_j$ are truly adjacent in GRN, then whether they affect each other's dropout, i.e., whether edges $Z_i \rightarrow R_j$ and $Z_j \rightarrow R_i$ exist, is naturally unidentifiable". This is because in this case, the (in)dependence between e.g., $Z_j$ and $R_i$ given at least $Z_i$, is untestable (see Proposition 3, Testability of CI queries).

Then, what if $Z_i$ and $Z_j$ are truly non-adjacent in GRN? Theorem 2 indicates that this non-adjacency, together with the absence of $Z_i \rightarrow R_j$ and $Z_j \rightarrow R_i$, can be identified barring one particular case. So,

> "What is this 'one particular case'"?

**R:** This particular case is when the non-adjacent $Z_i$ and $Z_j$ **always share a common child** (not descendant -- so we say it as so *particular*) that has to be conditioned on. This is formally defined in Proof of Theorem 2: "For every $\mathbf{S} \subset \mathbf{Z}$ that d-separates $Z_i$ from $Z_j$, there is an open path $Z_i\rightarrow W \leftarrow Z_j$, where either $W \in \mathbf{R}_{\mathbf{S}\cup \{i,j\}}$, or there exists a $k \in \mathbf{S}\cup \{i,j\}$ s.t. $W$ is $R_k$'s ancestor". Only in this particular case, the true d-separation cannot be recovered, resulting in a false adjacency.

We thank you for your detailed interest in these aspects of our work. For further clarity, we hope you will find Appendix C, particularly the illustrative examples (Figures 8 and 9), helpful.

Reviewer comment (Theory):
The paper does not mention *causal sufficiency*, which is an important assumption when using algorithms like PC and GES. Clarification is requested on whether it is reasonable for this type of data to assume there are no hidden confounders, and it is suggested to address this important practical limitation.

Author response:
**(Q4)** The reviewer suggests explicitly mentioning the causal sufficiency assumption, given its significance in the context of constraint-based methods.

**R:** We appreciate your emphasis on this. Indeed, causal sufficiency was implicitly included here in our assumption A1, as one of the "common assumptions needed for the asymptotic consistency of constraint-based methods". We have made it explicit in the [updated submission](https://openreview.net/pdf?id=gFR4QwK53h). Furthermore, we completely agree that it is an important future research problem -- note that the proposed zero deletion for CI correction is general and can be seamlessly incorporated into any existing constraint-based methods, and it would be interesting to see how this idea can be incorporated into existing local causal discovery methods, or methods that allow hidden confounders.

Reviewer comment (Theory):
The authors state "while a zero entry of $X_2$ may be noisy (i.e., may be technical), a non-zero entry of $X_2$ must be accurate, i.e., biological.". As far as I can tell, non-zero values might also be technical due to sequencing/mapping/algorithmic errors, correct?

Author response:
**(Q1)** The reviewer wonders whether non-zero values could also be "technical", i.e., subject to sequencing errors.

**R:** Thank you for this insightful point. As discussed in Appendix B.1, it is indeed possible that non-zero values are also influenced by technical errors, and that should be addressed separately from the general measurement error issue [1]. In this paper, we focus only on the dropout errors, mainly for the following two reasons:

 - Unlike general measurement errors that are ubiquitous in empirical sciences with a long history of research, dropout errors are **peculiar to scRNA-seq data** (in contrast to bulk RNA-seq data), owing to specific technical challenges like low RNA capture efficiency (Section 1). Our study thus focuses on this more specific and recent issue in the field.
 - Empirically, dropout errors in scRNA-seq data tend to be **more harmful** to downstream tasks than measurement errors [2]. Non-zero values, despite potential technical errors, are usually centered around the true expression levels, with errors manifested as relatively minor random noise. It is less likely for a true zero to be falsely reverse-transcribed into a non-zero cDNA count (out of no mRNA molecules). However, a true non-zero can be dropped out to zero, and this happens excessively, leading to a substantial distortion of the true gene expressions. This distortion is a more pressing concern, and thus forms the crux of this research focus.

Reviewer comment (Theory):
Supposing elevated dropout rates of, say 50%, the method relies on the fact that a conditioning variable can be found which "breaks" dependencies. Is this correct?

Author response:
**(Q2)** The reviewer wonders whether the method relies on the fact that non-zero values of conditioned variables can be found, even with an elevated dropout rate of, say 50%.

**R:** Yes, our method relies on it, as exactly outlined in assumption A5, which states that a sufficient sample size of conditioned variables remains after zero deletion. A5 is justified in the following aspects:

 - A5 is commonly satisfied in real scRNA-seq data, thanks to the inherent sparsity of GRNs. As a gene usually only interacts with a relatively small number of other genes, we usually don't need to condition on a large number of variables, resulting in larger remaining sample sizes. This is empirically supported by our **real data experiments (Figure 3)**.
 - A5 is also validated by a synthetic experiment with varying dropout rates (Figure 12 in Appendix D.1): even when the dropout rate is **as high as 70%**, the remaining sample sizes for CI tests are still considerably high to avoid large Type-II errors, leading to consistently better SHDs and skeleton F1-scores.
 - Additionally, it's worth highlighting the advantages of scRNA-seq technologies which can nowadays easily produce hundreds of thousands of samples within a single run. This innovation renders concerns over sample size somewhat mitigated.

Reviewer comment (Reproducibility):
The source code could be improved and properly documented (What libraries are required? Which versions of the libraries did you use for validation? How do I run all experiments? Etc etc.).

Author response:
**(Q4)** The reviewer recommends improving the clarity of presentations, such as in Figure 5 and in code documentation.

**R:** Thanks for raising this suggestion. In response, we have revised the explanation for Figure 5 in our [updated submission](https://openreview.net/pdf?id=gFR4QwK53h) for better clarity. Additionally, we have enriched the documentation in the [updated supplementary codes](https://openreview.net/attachment?id=gFR4QwK53h&name=supplementary_material) to provide more detail.

Thank you again for your positive feedback and valuable suggestions!