Paper: Doubly Robust Structure Identification from Temporal Data (ICLR-2024)

Reviewer comment (Evaluation):
Generally speaking, the paper is hard to follow and the goals of the proposed method are unclear given the entire paper. The approach is motivated as to be for (time-varying) causal feature selection for a target variable. However, it is claimed in the paper that it can also be used for full causal discovery, but it's not clear the evaluation is for either causal feature selection or full causal discovery.

Author response:
Thank you for your comment! If you have specific suggestions on how to improve the write-up, we will gladly incorporate your suggestions in our manuscript. 

To answer your question, in the experiments, we consider both causal feature selection and causal discovery. The reason why we consider both scenarios is to ensure a fair comparison with competitors. Note that causal feature selection for timeseries data is relatively underexplored. Hence, we compared our method with causal structure learning algorithms to showcase performance of our method with stronger evidence. In Section 6.1 we focus on causal discovery, whereas in Section 6.2 / Appendix J.1 we focus on causal feature selection. We are happy to make this point clearer in our submission.

Reviewer comment (Evaluation):
The task performed in the evaluation section is not described at all. Presumably the task should be causal feature selection, but all the reader is told is the metric used for evaluation is AUC, which doesn't sound like we're evaluating the correct set of causal features. Furthermore, the methods used in the evaluation section do not appear to be causal feature selection methods and are different from the related methods mentioned earlier in the paper.

Author response:
Yes, thank you for pointing it out! Our experimental framework essentially replicates previous related work. In our experimental section, we report on all state-of-the-art benchmarks for the DREAM3 datasets. Furthermore, the use of the AUC is consistent with previous related work [1] (state-of-the-art). There are several reasons for the usage of AUC other than other metrics like accuracy, F1 score, recall, etc, when comparing the performance of statistical algorithms that involve statistical tests. All of those metrics need a p-value thresholding which may lead to the following potential problems in the comparisons: vulnerability to class imbalance, task-specific threshold choices, model sensitivity to false positives and negatives, sensitivity to class probabilities, etc.

Reviewer comment (Evaluation):
Aside from the above confusion about the evaluation section, the empirical work is minimal in general and standard errors are not included.

Author response:
Yes, thank you for pointing it out. Our experimental framework considers a variety of baselines, and it is consistent with previous related work [1,2,3,4]. Throughout our experimental investigaion, we did not find that standard errors play a significant role, as it was also observed in related work [1]. Please inform us of other experiments/datasets/plots that you would think that our manuscript can benefit from, so we can incorporate your comments in our experiment section.

Reviewer comment (Evaluation):
Can the authors explain the evaluation task? Is it causal feature selection? What is the actual target that AUC is reported for?

Author response:
In the semi-synthetic experiment involving the Dream3 dataset (section 6.1, Appendix J.2), our objective is full causal discovery. Each of the five tasks within Dream3 involves observing the evolution of a system comprising 100 genes over time, subjected to 46 different perturbations. The ground truth causal graph delineating the relationships among these genes is provided. Out of the $100*99$ possible directed edges among the genes, some are indeed present, indicating direct causal influence, while others are not. This forms the actual target for AUROC calculations.

In the additional synthetic experiments (Appendix J.1), the focus shifts to causal feature selection. Here, the ground truth used in the AUROC metric is represented by the $\Delta \times m$ binary adjacency matrix $\Sigma^{\mathbf{Y}}$, where $\Delta$ signifies the lag, and $m$ represents the number of potential causes."

Reviewer comment (Evaluation):
How is the time-varying nature accounted for in the evaluation?

Author response:
In our experiments, we adopt the stationarity assumption alongside a fixed lag. To account for the time-varying nature of the dataset, we carefully select a suitable lag. It's important to highlight that while the stationarity assumption is included in the literature for experimental convenience [1,2,3,4], it is not a necessary condition for the application of our method.

Reviewer comment (Evaluation):
Why are standard errors missing from the baselines? Is the improvement significant?

Author response:
The baseline results in Table 1 are reported directly from [1,2], where the standard errors are also missing.

As noted in [1], RHINO is very stable and the standard error is minimal. This is also true for our propsed method. Hence, the improvement in our experimental framework can be considered significant. Please take into account the computational aspects that our method using simple regressions outperforms deep learning-based approaches that necessitate substantial computational resources, particularly GPUs.

Reviewer comment (Theory):
Is there a limitation when extending the approach to the full causal discovery setting?

Author response:
Our method naturally extends to full causal discovery as it is. In order for the proposed method to work, Axioms $(A)$ to $(C)$ must hold for any choice of the target variable Y. In the full causal discovery setting, this implies that there are no confounding effects or instantaneous effects between any of the time series of the model.

Reviewer comment (Theory):
The paper focuses on time series data but there is no statistical analysis focusing on time series data. For example, on page 7, it is stated that "Under mild conditions on the convergence of $g_j^0$, $g_j^i$ and $\alpha_j^0$, $\alpha_j^i$, the quantity $\theta^0 − \theta^i$ has $\sqrt{n}$-consistency" and that "We refer the reader to Chernozhukov et al. (2022; 2018) for a proof of the $\sqrt{n}$-consistency for estimates as $\theta^0$ and $\theta^i$." I do not think the cited references deal with time series data directly. It is disappointing that the paper does not provide any extensive treatment of time series analysis.

Author response:
Please note that, even though the data has time series structure, but i.i.d. trajectories of the data are provided and each trajectory can be treated as a single draw from an i.i.d. distribution. Hence, the strong consistency of our method follows directly from Chernozhukov et al. (2022; 2018). In our paper, we only provide an informal discussion on this point, to avoid a complicated notation that would have made the paper difficult to read. However, we are happy to provide a formal statment and an in-depth discussion of this point, if required. It is important to note that double ML can be extended to handle data of stochastic nature (Markovian non-i.i.d.) by applying it iteratively in a backward order, starting from the end of the process and moving towards the beginning. At each time step, following the doubly robust estimation, residuals of causal effects are computed to update the auxiliary random variables from the preceding time step. It can be demonstrated that the resulting scores derived from these auxiliary random variables satisfy the Neyman Orthogonality condition. This way the Markovian structure of the stochastic process at each step is cancelled out. The algorithm then proceeds to the previous time step. For further details, please refer to [1]. A similar approach can be employed here with appropriate adjustments, but delving into those modifications is beyond the scope of the main message conveyed in this manuscript.

Reviewer comment (Theory):
Lemma A.1 claims that conditional mean independence in part 1 is equivalent to the conditional dependence in part 2. This seems mainly driven by Axiom (A) where the error $\varepsilon$ is exogenous independent noise. I feel that this is a rather restricted setting. For example, suppose that Y is the time series of financial returns (e.g., S&P 500) and X is the causal factor that does not affect the conditional mean of returns but does affect the conditional variance of returns (typically called volatility in finance). It seems that the framework in the current paper excludes this kind of scenario. It is unclear to me what sense Axiom (A) is necessary; related to this point, Appendix A.3 is difficult to understand (see question 1 below).

Author response:
Axiom (A) essentially requires that no confounding effect exists between the target variable Y and the causal parents. This assuption is essential for identifiability. As you correctly observe, this setting is restrictive. However, without any assumption we simply cannot get identifiability. Conditional mean independence is a necessary assumption in this setting. In other words, if this assumption fails, as in the example that you provide, than no method will provably be able to recover the true underlying causal structure due to identifiability issues. Please refer to appendix A.3 for an example of this identifiability problem without Axiom (A).

Reviewer comment (Theory):
I cannot follow why equation (4) is a good property. This indicates that the bias multiplied by $\sqrt{n}$ goes to zero. It might be better to show that the root mean squared error multiplied by $\sqrt{n}$ goes to zero as $n \rightarrow \infty$. Some explanations would be helpful.

Author response:
Equation (4) is a standard strong convergence property, which follows from our analysis. This is indeed a good property, in the sense that the error between the true term and our approximation "quickly" converges to zero, for an increasing number of samples. Another way to express strong consistency is that as the sample size $n$ approaches infinity, $\sqrt{n}(\theta^0 - \theta^j)$ converges in distribution to $\mathcal{N}(0, \sigma^2)$, where the variance term $\sigma^2$ is independent of the sample size $n$.

Reviewer comment (Evaluation):
In the experiments on page 8, the area under the ROC curve (AUROC) is used as the performance metric. It would be beneficial why this metric is related to causality concerns.

Author response:
Regarding the experiments with Dream3, we mention that it is the standard practice to measure the performance of a new method on this benchmark using the AUROC metric.

This metric is applicable to binary classification tasks; all the predictions are sorted from most likely to be positive to least likely, and the AUROC measures the ability of the model to correctly rank these predictions.

In the Dream3 challenge, the task is gene network inference, so researchers assess how well their methods can identify true causal relationships amidst many potential but non-existent ones. Given the inherent binary classification nature of causal inference (an edge of the gene network either exists or not), the AUROC metric is more appropriate than other metrics (e.g. accuracy, F1 score) to distinguish between the two classes for this particular dataset. More specifically, Dream 3 is characterized by class imbalance; gene networks are very sparse. For example in the E. Coli 1 task of Dream 3, out of  the 100*99 potential directed edges between the 100 genes, less than 200 were present. AUROC is less sensitive to class imbalance compared to e.g accuracy, because the we simply sort the predictions about existence/absence of an edge according to how much confident our causal method is; that is, only the relative order matters. Furthermore, in this particular domain, a.k.a gene network inference, it's often more critical to correctly rank pairs of genes by their likelihood of having a causal relationship than to classify each pair definitively. AUROC evaluates a model's ability to rank positive cases higher than negative ones, which is directly aligned with this requirement.

Reviewer comment (Theory):
In the contribution, the authors claim that the proposed algorithm can be used for full causal discovery under some assumptions. The related discussion in section 5.2 is limited without details.

Author response:
Thanks a lot for point this out! This simple reduction was not included in the interest of space and it goes as the following:

If the underlying causal structure doesn't have hidden confounders (noted as "fully-observed" in the manuscript) and doesn't contain any cycles (noted as "acyclic" in the manuscript), then we can iteratively select each node of the causal graph as our target variable and apply our causal feature selction algorithm to find its direct causes. These two conditions are necessary for the Axiom (A) to be true when any node of the causal graph could be selected as the target variable. In the end, the whole causal structure will be discovered. 

We will make sure to clarify further in the final version of the manuscript.

Reviewer comment (Reproducibility):
In principle, the approach adheres to steps (1) through (4) in section 4.2, yet the practical algorithm has been adjusted to account for the time-consuming nature of "large instances." While the approach outline aligns with the proven theorem, a gap exists between the outlined approach and the modified algorithm. Is it feasible to implement the approach strictly in smaller instances, adhering to steps (1) through (4)? Furthermore, what does "large instances" imply in this context?

Author response:
Thanks a lot for your precise and detailed question! When we mention "large instances", we are specifically referring to cases where the number of nodes, denoted as $m$, is substantial.  The principled approach necessitates $\mathcal{O}(m)$ regressions, which is linear in the number of nodes. However, by incorporating the zero-masking heuristic, we can optimize the process, reducing the required regressions to just $\mathcal{O}(1)$. This proves particularly advantageous in situations where computational resources are limited, but the causal graph is huge. It's essential to highlight that, as demonstrated in Appendix A.7, there is no discernible difference between the principled approach and the one utilizing the heuristic when the underlying estimator is linear regression.

Reviewer comment (Experiments):
There is no real-world application provided in the paper.

Author response:
Indeed, thank you for highlighting this consideration. In previous related works [2,3,4,5], Dream3 was employed as a gold-standard benchmark which is of systems' biology nature. However, we acknowledge that over-tuning causal discovery algorithms exclusively for a single dataset could present potential issues. As you aptly observe, other datasets might be pertinent in this context. We are open to the inclusion of any additional datasets you may suggest.

Reviewer comment (Theory):
Can you please provide a brief explanation of the role played by the causal graph in the proposed algorithm? Personally, I am under the impression that the causal graph is unrelated to the proposed method, rendering the faithfulness assumption, cyclic structure, and causal sufficiency irrelevant to the algorithm. Thus, I do not consider the relaxation of this assumption as an advantage of the method, as it falls outside the scope of the algorithm. Please correct me if I missed the point.

Author response:
Our algorithm uncovers the underlying causal structure of the DGP. Hence, the causal graph is very related to how the algorithm works, in the sense that a different causal structure will result into a different outcome for the model. Without some assumptions on the nature of the DGP, however, the proposed statistical test will not uncover the true causal structure, and the results will be inconclusive. The assumptions that you mention are very relevant to interpret the results of the algorithm, even if they do not explecitly appear in the pseudo-code.

Reviewer comment (Writing):
In the first equation on page 4, what is $N$?

Author response:
Thank you for spotting this typo! In fact, consistent with the notation used in the corresponding section of our paper, the correct equation for NAVAR is $$Y_T = \beta + \sum_{i=1}^{m} f_{i} (X^{i}_{T - \kappa:T-1}) + \eta_T, $$ where $m$ is number of nodes.

Reviewer comment (Writing):
In equation 3, what is $n$? Should it be $k$?

Author response:
$n$ is the number of samples. The equation 4 is about the strong consistency statement. Another way to represent that is by  $(\theta^0 - \theta^i) \overset{\mathrm{d}}{\rightarrow} \mathcal{N}(0, \frac{\sigma^2}{n})$ (where $\overset{\mathrm{d}}{\rightarrow}$ denotes convergence in distribution), indicating that the variance of the estimation error tends to zero as the sample size ($n$) approaches infinity.

Reviewer comment (Writing):
I felt lost that in equation 3, $g^0_0$ and $g^i_0$ equal to the same conditional expectation as $\alpha^0_0$ and $\alpha_0^i$ in the second point in section 5.1. Are they the same things?

Author response:
Yes that's correct! This is just a nice coincidence of the application of the doubly robust score $\varphi( \theta, \boldsymbol \eta) := m(\boldsymbol{V}; g ) + \alpha(\boldsymbol{X})\cdot(Y - g(\boldsymbol{X})) - \theta$ (Equation 2 in the paper) to our particular setting, where the linear moment functional takes the form $m(\boldsymbol{V}; g )=Yg$ (proof of theorem 4.2 in the Appendix). Consequently, the Riesz Representer of this specific functional is $\alpha(\boldsymbol{X})=\mathbb{E}[Y\mid \boldsymbol{X}]$. Actually, in our experiments we have used explicitely this fact and since the Riesz representer $\alpha(\boldsymbol{X})$ coincides with the regressor $g(\boldsymbol{X})$ we can train only one model (we have ablated on this matter, i.e training two separate models for $\alpha(\boldsymbol{X})$ and $g(\boldsymbol{X})$, or just use a single model for both, and experimentally found out that there was no difference).

However, in general the Riesz Representer $\alpha(\boldsymbol{X})$ does not coincide with the regressor $g(\boldsymbol{X})$; it does not even have an elegant explicit closed form. Moreover, the way of calculating it can be very different; for example, in [1],it is the minimizer of a particular loss, the Riesz Loss: $\widehat{\alpha} = argmin_{\alpha\in {\cal A}_n} \mathbb{E}_n[\alpha(Z)^2 - 2m(W; \alpha)]$.

Since in practice $\alpha(\boldsymbol{X})$ and $g(\boldsymbol{X})$ are separate quantities, and in order to avoid confusion, we use different notation.

Reviewer comment (Experiments):
As the appendix states, $k$ ranges from 3 to 7. What is the value of $k$ used in each experiment? Is the algorithm output sensitive to the value of $k$?

Author response:
In all of the experiments reported in the paper, we have set k=5, with the only exception of figure 3 in the Appendix (page 29), where we used k=10. In this last experiment, where we test the performance of our algorithm in the low data regime, there are some settings where only as few as 5 (out of 46 in total) trajectories are used, so the number of observations is quite limited. In that case, we experimentally observed it is better to use k=10 folds (i.e each model training uses 90% of the data) than k=5 folds (i.e each model training uses 80% of the data). In all other settings, we experimentally observed that using k=10 yields the same performance compared to k=5. Moreover, we also found k=5 outperforms k=3.

But even in the cases where some configuration for k is better compared to another one, the difference in performance (of the AUROC metric) is usually less than 5%. So the output is not very sensitive to the value of k.

Reviewer comment (Presentation):
The term "trajectories" means the time series, correct? In Fig.2, what does $N_feat$ represent? Is $N_feat$ indicative of the number of trajectories? Additionally, in Fig.3, all the algorithms exhibit improved performance with an increase in the number of trajectories. Could you provide a brief explanation for this trend? Moreover, why does Fig.3 depict the performance in low-sample regimes, and how are "low-sample regimes" reflected in the Fig.3?

Author response:
We use the term "_trajectories_" only when we refer to the Dream3 dataset. In that context, _trajectory_ means group (i.e set) of time series. More specifically, the Dream3 challenge consists of 5 different tasks, where for each task we are given 46 groups of time series, and each group consists of 100 time series, each time series having length 21 timesteps. In this dataset, for a specific task, e.g E. Coli 1, we see how the measurements of 100 genes varie in time, after a perturbation to the system has occured at timestep t=0. In total, 46 different perturbations are applied to this system of 100 genes. Importantly, the causal dynamics of the gene system is the same in all these 46 different trajectories.

In Fig.2, $N_{feat}$ actually is the same as $m$, the number of potential causes.

Regarding Fig.3, based on the above explanation of trajectories, it is clear that fewer trajectories implies smaller number of observations used for causal inference. For example, when we have  5 trajectories, this means that we have observed the system of 100 genes in 5*21=105 timesteps in total, while 46 trajectories means that we have observed the system of 100 genes in 46*21=966 timesteps in total.

Reviewer comment (Experiments):
Is there a specific reason for using only one baseline algorithm in the experiments presented in the appendix?

Author response:
We only compare agains RHINO as this baseline is the main competitor and the best-performing algorithm. Other baselines are expected to perform worse, as already shown in previous related work and the results in Dream 3 benchmark (section 6.1).

Reviewer comment (Theory):
Why does the proposed method allow the existence of cycles or hidden confounding?

Author response:
Thank you very much for taking the time to review our work and recognize its merits!

> The paper did not analyze or give an intuition why the proposed method allows the existence of cycles or hidden confoundings. Why does the proposed method allow the existence of cycles or hidden confounding?

To give an intuition of why we can handle hidden confounding and cycles, denote with $Y$ the target time series and with $\{ X^1, \dots, X^n \}$ the observed time series that are potential causes for $Y$. Using Axioms $A$-$C$, we prove that in this case, performing an intervention on all the variables  $\{ X^1, \dots, X^n \}$ is equivalent to conditioning (this fact is specific to our causal model and it is not true in general). This type of conditioning "cancels out" the effect of cycles or hidden confounding on $\{ X^1, \dots, X^n \}$.

Reviewer comment (Theory):
How do you identify cyclic structures? Does Algorithm 1 have the capability to detect cyclic structures, and does this imply the presence of confounders?

Author response:
Our algoirthm only identifies direct causal effects on a target variable of interest. It does not identify cycilc structure. However, as detailed in Axioms (A)-(B), our algorithm works even under confounding or cycles among the covariates. In other words, even if hidden confounders and cycles are present, we are still able to uncover the direct causes.

Reviewer comment (Reproducibility):
In your method, is the time lag $k$ fixed, or does it remain stationary but vary among different variables?

Author response:
In general, we only assume stationarity, and $k$ may vary among different variables. However, in our experimental framework we set $k$ to always be the same for simplicity. Specifically, in all our experiments, time lag is equal to 2. This is quite a standard value accross the literature [1,2,3,4].