Paper: Learning Large DAGs is Harder than you Think: Many Losses are Minimal for the Wrong DAG (ICLR-2024)

Reviewer comment (Theory):
My main concern is the practicality of the results for the non-linear case; see Questions. I'd be happy to consider increasing my score if the authors offer compelling arguments against my concerns.

Author response:
Thank you for sharing your insightful feedback on our work! We appreciate that you agree with us regarding the importance of our findings. We will of course we will fix all the minor issues you've pointed out, thanks for that! We answer your major concerns below.

> Could the authors explain how these results relate to the identifiability results of Peters et al. [1, Cor. 31]? They state that the ground truth graph is identifiable assuming that each f is three times differentiable and non-linear, so what is then the motivation of considering MMSE in the non-linear case if it can lead to an incorrect graph as the authors describe?

Thank you for sharing this interesting thought! In fact, **we share your concerns** in using log-likelihood based scores (or MMSE-like scores) for structure learning as we explain subsequently: 

Indeed, Peters et. al. showed that under the assumption of  three times differentiable, non-linear functions and additive Gaussian noise even the *causal* graph structure is identifiable from a distribution. However, note that the result reported in Peters et. al. are theoretical conditions under which the true DAG does not violate any of the assumptions made (non-linearity in each argument of each function $f_i$ and Gaussian noise) while any other possible DAG would violate these assumptions. 

In practice, such a graph is learned by comparing different DAGs and test whether they violate the assumptions or not. Since the number of possible DAGs grows super-exponentially in the number of nodes, one usually only compares DAGs within the same Markov Equivalence Class (MEC) to make the problem feasible, i.e. the set of possible solutions is reduced to graph that share the same independence-structure. 
Identifying the MEC in turn is equivalent to the structure learning tasks our work is considering, i.e. constructing a graph encoding all independencies of the data generating process correctly. However, the methods we considered in our work (except for GES) are supposed to yield one DAG from the correct MEC.
Identifying the correct MEC or a DAG from a MEC is often done by optimizing log-likelihood (which degenerates to MMSE-like scores under Gaussian noise) due to the following reasons: 
1. The correct independence structure (i.e. MEC) should be identified if the model found maximizes log-likelihood. This is because wrong independencies would imply a wrong factorization of the corresponding distribution, i.e. yield a worse log-likelihood score. 
2. Log-likelihood is usually efficient to compute (e.g. additive Gaussian noise case)

Our work shows that under certain conditions log-likelihood is maximized for DAGs encoding wrong independence properties. Hence we share your concerns about using such scores for structure learning tasks.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

Reviewer comment (Theory):
Could the authors explain how these results relate to the identifiability results of Peters et al. [1, Cor. 31]? They state that the ground truth graph is identifiable assuming that each f is three times differentiable and non-linear, so what is then the motivation of considering MMSE in the non-linear case if it can lead to an incorrect graph as the authors describe?

Author response:
Thank you for sharing your insightful feedback on our work! We appreciate that you agree with us regarding the importance of our findings. We will of course we will fix all the minor issues you've pointed out, thanks for that! We answer your major concerns below.

> Could the authors explain how these results relate to the identifiability results of Peters et al. [1, Cor. 31]? They state that the ground truth graph is identifiable assuming that each f is three times differentiable and non-linear, so what is then the motivation of considering MMSE in the non-linear case if it can lead to an incorrect graph as the authors describe?

Thank you for sharing this interesting thought! In fact, **we share your concerns** in using log-likelihood based scores (or MMSE-like scores) for structure learning as we explain subsequently: 

Indeed, Peters et. al. showed that under the assumption of  three times differentiable, non-linear functions and additive Gaussian noise even the *causal* graph structure is identifiable from a distribution. However, note that the result reported in Peters et. al. are theoretical conditions under which the true DAG does not violate any of the assumptions made (non-linearity in each argument of each function $f_i$ and Gaussian noise) while any other possible DAG would violate these assumptions. 

In practice, such a graph is learned by comparing different DAGs and test whether they violate the assumptions or not. Since the number of possible DAGs grows super-exponentially in the number of nodes, one usually only compares DAGs within the same Markov Equivalence Class (MEC) to make the problem feasible, i.e. the set of possible solutions is reduced to graph that share the same independence-structure. 
Identifying the MEC in turn is equivalent to the structure learning tasks our work is considering, i.e. constructing a graph encoding all independencies of the data generating process correctly. However, the methods we considered in our work (except for GES) are supposed to yield one DAG from the correct MEC.
Identifying the correct MEC or a DAG from a MEC is often done by optimizing log-likelihood (which degenerates to MMSE-like scores under Gaussian noise) due to the following reasons: 
1. The correct independence structure (i.e. MEC) should be identified if the model found maximizes log-likelihood. This is because wrong independencies would imply a wrong factorization of the corresponding distribution, i.e. yield a worse log-likelihood score. 
2. Log-likelihood is usually efficient to compute (e.g. additive Gaussian noise case)

Our work shows that under certain conditions log-likelihood is maximized for DAGs encoding wrong independence properties. Hence we share your concerns about using such scores for structure learning tasks.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

Reviewer comment (Novelty):
While the implications of the statements are definitely important, they appear rather obvious, since, for instance, it is well known that the L2 loss is closely connected with the variance. In that sense, the issue with the scaling is already discussed in other works. However, that being said, I did not find a paper that particularly focuses on this issue in a self-contained manner.

Author response:
Thank you for appreciating the importance of our work! We are aware of other works showing or indicating similar problems of e.g. the L2 loss (see [1, 2]). However, these works so far lacked general statements such as our finding that **all** log-likelihood based losses share susceptibility towards scaling in the context of structure learning. Further, we extended existing results to $d$-dimensional cases, hence our results are far more impactful for practical applications. Works like [1, 2] focused on either bi-variate cases, special losses like L2 or both.

Reviewer comment (Theory):
The need for A2 remains unclear. In particular, the "invertibility" of the "function" in the related literature typically refers to the invertibility with respect to the noise, not with respect to the input. This is, in a general SEM Y = f(parents, noise), the f needs to be invertible with respect to the noise, but it doesn't matter for the parents. Otherwise, this would be an extremely strong assumption. In the case of additive noise models, this invertibility is always given by definition, since N = Y - f(parents), i.e., it is not a restriction on f. However, I might have also missed something here and maybe the authors can clarify this.

Author response:
We thank you for pointing this out! Of course the invertibility only needs to hold w.r.t. the noise. We corrected this in our paper and made Assumption 2 clearer in that regard. However, please note that this does not affect our general theoretical results as A2 is only a requirement for encoder-decoder based structure learners to allow reconstruction.

Reviewer comment (Novelty):
While you reference great related literature commenting on the same issue, it remains unclear where they lack and where you are filling the gap.

Author response:
Let us try to clarify our contributions better: Current works as [1, 2] only discuss similar results for either (1) special loss functions like L2, (2) bi-variate cases, (3) linear dependencies among variables or any combination of these. We fill this gap by presenting clear theoretical results showing that (1) the family of log-likelihood based losses is not suitable for structure learning in (2) $d$-dimensional, (3) non-linear cases. Please note, that we stated our contributions in the Introduction which we now highlighted better in the text:

Consequently our contribution will (1) generalize the above mentioned results to $d$-dimensional and non-linear cases, (2) provide exact conditions under which MMSE fails to identify the correct structure and (3) show that all log-likelihood based losses are susceptible to scaling under appropriate assumptions, both theoretically and empirically.

Reviewer comment (Novelty):
The proposed approach in Section 3.3 seems rather trivial. Here, I think the main focus of the paper should rely on the theoretical statements.

Author response:
We agree that the main focus of our paper are the theoretical and empirical results. Section 3.3 is intended as a starting point to tackle the problems of log-likelihood based losses in the context of structure learning and to encourage further research on this. To make this clearer, we have adapted Sec. 3.3 accordingly.

Reviewer comment (Theory):
It would be very insightful to comment more on the assumption A1, since it seems to be very strong. For instance, the formulation of A1 implies that you only consider a chain or a fork or a collider, but none of the combinations. If this is true, this needs a particular remark and justification. Otherwise, it does not seem to be useful for any realistic graph structure beyond these trivial ones.

Author response:
You raise an important point and we agree that an additional remark is beneficial for clarity. In our theoretical analysis we indeed assume chains, forks or colliders. However, note that any DAG is composed of these three structures, hence our results still hold for any arbitrary subgraph where A1 is fulfilled. 
In our point of view, the more important point about A1 is that it implies that each node in a DAG cannot have two roles, e.g. a collider's sink and a fork's origin. Indeed, such cases are not covered by our theoretical findings and are an interesting path for further investigation. We clarified this with an additional remark on A1 in the paper.

Reviewer comment (Theory):
In this regard, at the end of paragraph 3, you say that with A1 you consider "all possible substructures", but A1 is clearly defined as "either .." and not a combination.

Author response:
What we mean here is that our theoretical results cover all possible substructures a DAG can be composed of except when a node has multiple roles as described above. We clarified this by saying that if A1 holds, each DAG we consider can be decomposed into a set of sub-graphs consisting of chains, forks and colliders.

Reviewer comment (Theory):
Assumption A2 is a bit confusing, since typically, one only requires invertibility with respect to the noise and not the functional relationships (as mentioned in the 'Weakness' section). For instance in DAG-GNN, the "invertibility" only refers to the invertibility of Y = f(X, N) with respect to N. Similar to the previous point, the need for A2 requires more justification and explanation, since it appears to be a very strong assumption in the current form.

Author response:
We thank you for pointing this out! Of course the invertibility only needs to hold w.r.t. the noise. We corrected this in our paper and made Assumption 2 clearer in that regard. However, please note that this does not affect our general theoretical results as A2 is only a requirement for encoder-decoder based structure learners to allow reconstruction.

Reviewer comment (Theory):
While I understand the focus on the MMSE, it should be noted that in an optimization task, some regularization is required. Otherwise, the best solution for ||X - f(X)||_2 is f(X) = X, the identity. Here, a brief remark on this would be insightful to avoid confusion.

Author response:
We fully agree on that and now state explicitly that we only consider adjacency matrices representing a DAG in the beginning of Sec. 3. This is the same as assuming that the optimization procedure is capable of enforcing the constraints represented by regularization terms (e.g. the resulting graph is a DAG).

Reviewer comment (Theory):
In Section 3.3, the exclusion of "free variance terms" needs more clarification. It currently reads as that one needs to know the graph structure to identify these terms, while the whole task is to infer the graph structure in the first place. I assume there is a misunderstanding where the authors can maybe comment on.

Author response:
Thanks for helping us to clarify this point. Please note that we explicitly state that our proposal of excluding free variance terms from the score only works for discrete optimization algorithms that iterate over candidate graphs (such as GES): 

"Consequently, we propose a straightforward protocol for enhancing the resilience of discrete structure learning algorithms, such as the GES algorithm:"

Here, the structure of each candidate graph is known and thus the free variance terms can be excluded from scoring.

Reviewer comment (Theory):
In this article, MMSE seems to be used for prediction, rather than for learning causal graphs. The definition of MMSE is the sum of squares of predicted residuals, which characterizes the statistical correlation of variables rather than causal relationships. It is an obvious conclusion that only using statistical correlation for causal graph learning will lead to errors.

Author response:
We agree that the MMSE characterizes only the quality of some model in terms of how well it fits statistical correlations. However, please note our work considers *structure learning* algorithms which are **not** designed to identify **causal** structures. Rather, these algorithms are supposed to identify **independence** structures within the data at hand. Please note that we explicitly state this in the Introduction section.

Reviewer comment (Theory):
In structure learning algorithms, the score function usually contains penalty terms, and the variable that needs to be optimized in the score function is the adjacency matrix W, rather than data X. In addition, the adjacency matrix W will also be included in the penalty term. So, why does the score function directly degenerate into a square based loss function? Or, why is optimizing the score function equivalent to optimizing MMSE?

Author response:
Thank you for pointing this out! We agree that scores used for structure learning usually contain penalty terms, e.g. to ensure that the learned graph is a DAG. In our theoretical analysis we assumed for simplicity that the constraints represented by these penalty terms (e.g. graph is a DAG) are fulfilled. Based on that we show that within the space of valid solutions (according to the constraints), structure learners using log-likelihood still may identify wrong structures under certain conditions. We stated this limitation in our work.
Please note that in our empirical evaluation these penalty terms were **not** excluded, underlining that our theoretical results are still of high relevance in practice.
Please also note that Proposition 7 shows that any log-likelihood based score is proportional to the MMSE score under Gaussian noise assumption.

Reviewer comment (Novelty):
In practical applications, most of the score functions we choose are not affected by data normalization, just like the SRL function mentioned in this article. Thus, this article does not have a clear critical objective, or rather, the critical target seems to be rare in practical applications, which greatly reduces the significance of this study.

Author response:
This is an interesting point, however we do not agree that most score functions chosen in practice are not affected by data normalization. In fact, our work clearly shows that the entire log-likelihood family under Gaussian noise assumption is affect by scale (see Proposition 8). To the best of our knowledge, most score based structure learning algorithms use such a score as an objective (e.g. GES [1], NOTEARS, [2] DAG-GNN [3], GranDAG [4]). Therefore we believe that our contribution is still significant, also for practitioners.

Reviewer comment (Novelty):
I think the biggest issue here is a lack of discussion of / contextualization with constraint based approaches.

Author response:
Thank you for your insightful feedback and we highly appreciate that you agree with us that research on limitations of log-likelihood based scores in score based structure learning is of high importance. We answer your concern below.

> Not necessarily a question, but it would be useful if the authors can add a discussion and / or empirical comparison of constraint based discovery methods.

We agree that a brief discussion on how constraint based structure learners are affected by our results is useful to contextualize our work properly. Therefore we added a discussion on that in our paper, stating that - **as long as the used (conditional) independence test is invariant to scaling the data** - constraint based methods are not affected by our findings. Instead of optimizing a score, constraint based algorithms test for (conditional) independencies among variables and use this information to construct a graph. We performed experiments that confirm our conjecture using PC-algorithm [1] and Grow-Shrink (GS) algorithm [2]. For each algorithm we used the Fisher-Z test.

Our empirical results show that neither PC, nor GS is affected by setting variables to a different scale. The following table shows how often (within 10 repeats) GS predicted a different graph if the data scale changes. The same results hold for PC.

|          | chain | fork | collider |
|--------|-----|----|--------|
|   chain  |   0%  |  0%  |    0%    |
|   fork   |   0%  |  0%  |    0%    |
| collider |   0%  |  0%  |    0%    |

We will add the empirical results on constraint based methods in the Appendix.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

[1] Kalisch et. al. Estimating High-Dimensional Directed Acyclic Graphs with the PC-Algorithm. JMLR. 2007.

[2] Margaritis . _Learning Bayesian Network Model Structure from Data_. Ph.D. thesis, School of Computer Science, Carnegie-Mellon University, Pittsburgh, PA. 2003.

Reviewer comment (Experiments):
It would also have been good to see comparisons to constraint based discovery in the experiments.

Author response:
Thank you for your insightful feedback and we highly appreciate that you agree with us that research on limitations of log-likelihood based scores in score based structure learning is of high importance. We answer your concern below.

> Not necessarily a question, but it would be useful if the authors can add a discussion and / or empirical comparison of constraint based discovery methods.

We agree that a brief discussion on how constraint based structure learners are affected by our results is useful to contextualize our work properly. Therefore we added a discussion on that in our paper, stating that - **as long as the used (conditional) independence test is invariant to scaling the data** - constraint based methods are not affected by our findings. Instead of optimizing a score, constraint based algorithms test for (conditional) independencies among variables and use this information to construct a graph. We performed experiments that confirm our conjecture using PC-algorithm [1] and Grow-Shrink (GS) algorithm [2]. For each algorithm we used the Fisher-Z test.

Our empirical results show that neither PC, nor GS is affected by setting variables to a different scale. The following table shows how often (within 10 repeats) GS predicted a different graph if the data scale changes. The same results hold for PC.

|          | chain | fork | collider |
|--------|-----|----|--------|
|   chain  |   0%  |  0%  |    0%    |
|   fork   |   0%  |  0%  |    0%    |
| collider |   0%  |  0%  |    0%    |

We will add the empirical results on constraint based methods in the Appendix.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

[1] Kalisch et. al. Estimating High-Dimensional Directed Acyclic Graphs with the PC-Algorithm. JMLR. 2007.

[2] Margaritis . _Learning Bayesian Network Model Structure from Data_. Ph.D. thesis, School of Computer Science, Carnegie-Mellon University, Pittsburgh, PA. 2003.

Reviewer comment (Experiments):
Not necessarily a question, but it would be useful if the authors can add a discussion and / or empirical comparison of constraint based discovery methods.

Author response:
Thank you for your insightful feedback and we highly appreciate that you agree with us that research on limitations of log-likelihood based scores in score based structure learning is of high importance. We answer your concern below.

> Not necessarily a question, but it would be useful if the authors can add a discussion and / or empirical comparison of constraint based discovery methods.

We agree that a brief discussion on how constraint based structure learners are affected by our results is useful to contextualize our work properly. Therefore we added a discussion on that in our paper, stating that - **as long as the used (conditional) independence test is invariant to scaling the data** - constraint based methods are not affected by our findings. Instead of optimizing a score, constraint based algorithms test for (conditional) independencies among variables and use this information to construct a graph. We performed experiments that confirm our conjecture using PC-algorithm [1] and Grow-Shrink (GS) algorithm [2]. For each algorithm we used the Fisher-Z test.

Our empirical results show that neither PC, nor GS is affected by setting variables to a different scale. The following table shows how often (within 10 repeats) GS predicted a different graph if the data scale changes. The same results hold for PC.

|          | chain | fork | collider |
|--------|-----|----|--------|
|   chain  |   0%  |  0%  |    0%    |
|   fork   |   0%  |  0%  |    0%    |
| collider |   0%  |  0%  |    0%    |

We will add the empirical results on constraint based methods in the Appendix.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

[1] Kalisch et. al. Estimating High-Dimensional Directed Acyclic Graphs with the PC-Algorithm. JMLR. 2007.

[2] Margaritis . _Learning Bayesian Network Model Structure from Data_. Ph.D. thesis, School of Computer Science, Carnegie-Mellon University, Pittsburgh, PA. 2003.