Paper: Constraint-Free Structure Learning with Smooth Acyclic Orientations (ICLR-2024)

Reviewer comment (Novelty):
Although the paper claimed several times that it provides "the first unconstrained optimization approach that learns DAG from observational data", several existing works have done so, such as https://arxiv.org/abs/2006.10201 and DP-DAG (https://arxiv.org/abs/2203.08509). DP-DAG has been discussed in Section 2, but it may not be clear why the paper still claims being "first". Also, it would be great to compare these methods in the experiments.

Author response:
**Contribution Claim.** In line with yours and other reviewers' comments, we have adjusted our claim. Further, as stated in the Related Works, we only considered for our empirical evaluation approaches that ensure the acyclicity of the solution, while GOLEM requires to iteratively threshold the solution to ensure acyclicity. Given the reviewer's suggestion, we included in Appendix E.17 an empirical comparison with GOLEM to further motivate our claim. Concerning the comparison with DP-DAG, even if possibly less apt for practical applications, simpler and scalable optimization schemes like ours are more suitable to be integrated as components in continuous causal discovery approaches. Given the reviewer's concern, we further clarified the issue in the Related Works and reported additional experiments in Appendix E.19. The additional experiments report results for DAGuerrotype (Zantedeschi et al., 2023), a similar but faster method comparable to DP-DAG, which however already highlights significant scalability issues when compared to optimization-schemes such as COSMO.

Reviewer comment (Evaluation):
It may not be clear how to interpret AUC as compared with NHD. For instance, for the results of 500 nodes in Appendix E.3, the NHD of COSMO and DAGMA are 3.57 and 1.73 respectively, indicating that COSMO performs much worse than DAGMA (probably due to lots of false discoveries). However, the AUC of COSMO is slightly better than DAGMA.

Author response:
**AUC, NHD, and Acyclicity** By following NOTEARS, most methods perform a thresholding step to prune off small weights and spurious arcs (Ng et al., 2023). To compare with them we adopted the same practice to report NHD, which can be computed only on binary matrices. Overall, we consider AUC to be a more suitable metric since it does not require thresholding the matrix and better captures the tradeoff between true positive and false positive ratio. Notably, the same reasoning underlies the metric chosen for validating DP-DAG in Charpentier et al. (2022). Concerning the reviewer's question, our solution is guaranteed to be acyclic before the thresholding step. To further back this claim, we updated the manuscript with a new section in Appendix G, which reports the trend of temperature and acyclicity and shows how the model reaches acyclicity zero even for non-zero temperatures.

Reviewer comment (Evaluation):
I would suggest reporting NHD in the main paper that might be more commonly used in structure learning (e.g. see DAGMA and NOTEARS papers).

Author response:
**AUC, NHD, and Acyclicity** By following NOTEARS, most methods perform a thresholding step to prune off small weights and spurious arcs (Ng et al., 2023). To compare with them we adopted the same practice to report NHD, which can be computed only on binary matrices. Overall, we consider AUC to be a more suitable metric since it does not require thresholding the matrix and better captures the tradeoff between true positive and false positive ratio. Notably, the same reasoning underlies the metric chosen for validating DP-DAG in Charpentier et al. (2022). Concerning the reviewer's question, our solution is guaranteed to be acyclic before the thresholding step. To further back this claim, we updated the manuscript with a new section in Appendix G, which reports the trend of temperature and acyclicity and shows how the model reaches acyclicity zero even for non-zero temperatures.

Reviewer comment (Experiments):
It seems that there is much more performance drop for nonlinear case. Is there a reason for not including NOTEARS-MLP for 40 and 100 nodes?

Author response:
**Non-Linear.** We agree with the reviewer that there is indeed a performance drop in the non-linear scenario, which is however shared by the baselines and mainly due to the increased difficulty of the task itself. We did not include NOTEARS-MLP for larger graphs because of the significantly higher computational time required for already 20 nodes in the non-linear scenario.

Reviewer comment (Novelty):
(major) The paper neglects the order-based literature for DAG learning which is closely related to this work. Only VP-DAG (Charpentier et al. 2021) is mentioned, but a considerable line of research exists, starting from the seminal works [1,2], to the most recent ones [e.g. 3,4] based on permutation matrices, and in particular [5] that also makes use of a priority vector. The paper does not highlight the pros and cons wrt this literature nor compare with any of the order-based methods empirically.

Author response:
**Related Works.** Given your suggestions and the comments by other reviewers we have carefully enriched the related works section, which is now available in the updated version of our manuscript. Overall, we propose a parameterization and an optimization strategy to continuously model the space of DAGs, strictly following the line of work of NOTEARS. On the other hand, both DP-DAG (Charpentier et al. 2021) and BCD-Nets (Cundy et al. 2021) are Bayesian causal discovery algorithms that learn a distribution over DAGs given observational data. Similarly, DAGuerreotype (Zantedeschi et al. 2023) retrieves a DAG by fitting a distribution over permutations. We added and discussed the necessary references to clarify the scope of our work to the readers. For this reason, while our main empirical evaluation still focuses on other optimization schemes, we also added in Appendix E.19 empirical results of DAGuerreotype on the same experimental testbed. Our results highlight that DAGuerreotype does not scale on the size of the graph as simpler optimization schemes, which could be then more easily integrated as components in other continuous causal discovery approaches.

Reviewer comment (Experiments):
(major) Experiments are carried out on synthetic data with weight settings that are known to have trivial solutions [Reisach et al. 2021]. Hence they are insufficient to validate the method. The experimental analysis should be carried out on more challenging settings (e.g., smaller edge weights) where the trivial baseline sortnregress [Reisach et al. 2021] is not state-of-the-art.

Author response:
**Evaluation.** As we remarked in the main body, we are aware that continuous approaches are particularly susceptible to data normalization and might exploit variance ordering between variables, as shown by Reisach et al. 2021 and originally by Loh and Bühlmann (2014). Therefore, empirical results on simulated datasets that do not explicitly control this condition might not generalize to real-world scenarios. However, we propose a loss-agnostic optimization scheme that aims at being integrated as a building block for causal applications as it is the case for NOTEARS, DAGMA, or NOCURL. By comparing on the exactly the same test bed of these methods, we show that at parity of conditions with the existing literature on acyclic optimization we get comparable results for significantly less time. While surely interesting as a future direction, defining function classes, appropriate losses, and identifiability results are characteristics of a causal discovery method, which is out of the scope of the current work. However, the suggestion by the reviewer is indeed extremely valuable to match our remarks with experimental results. Therefore, we include in Appendix E.18 a comparison for our and the remaining models on variance normalized data. Our results highlight how performance degradation is mostly due to the choice of MSE and does not depend in general on the particular optimization strategy, as COSMO is affected comparably to all competing methods.

Reviewer comment (Theory):
By using the tempered sigmoid at training, the adjacency matrix is never acyclic. Acyclicity is only obtained in the limit of the temperature tending to 0, that however turns the gradients to 0. This implies that careful temperature scheduling is required in order to converge to a good solution.

Author response:
**Temperature.** We agree that temperature scheduling is fundamental in our solution. In practice, as we describe in the methodological section, we simply apply cosine annealing from $t=0.45$ to a relatively small temperature during training. In practice, because of the introduced regularization, the acyclicity of the solution consistently drops before reaching zero. For a visualization of this behavior please refer to the newly introduced Appendix G, where we report the trends of acyclicity and temperature. In particular, in the last phase of training, for non-zero temperature the acyclicity already reaches zero and both the directed and the smooth orientation are still optimized in this "acyclic regime".

Reviewer comment (Reproducibility):
The method relies on edge thresholding at test time to remove spurious relationships. However a method for choosing such threshold is not presented.

Author response:
**Thresholding.** While L1 regularization already encourages sparsity, all mentioned methods prune weights with absolute values smaller than 0.3, which is a de facto standard threshold used by NOTEARS, DAGMA, NOCURL, GOLEM, and others (Ng et al. 2023). While we do not delve into this matter, we report in our experimental comparison how existing work criticizes this practice, e.g., Xu et al. (2022). Further, we emphasized in the main body the AUC metric, since it does not depend on a fixed threshold and allows us to evaluate discovery for a varying threshold. Notably, DP-DAG shares the same choice of metric.

Reviewer comment (Experiments):
The results in the main text are still exclusively on possibly flawed settings. Because of this, these results show that the proposed method is much faster than the considered baselines, but they are not conclusive regarding its ability to perform structure learning. The authors added results on normalized setting in the appendix, however without reporting running times and only for a subset of the noise and graph configurations. The additional comparison with sortnregress, that would highlight the triviality of certain settings, is also reported separately from the main results and it again covers only a subset of the noise and graph configurations.

Author response:
**Evaluation.** We included in Appendix E.18 the results of "sortnregress" on several of our settings. Since we share the exact same experimental testbed of NOTEARS, DAGMA, and NOCURL, the "sortnregress" baseline performs better on some settings as expected, while notably being slower than our approach on larger graphs. With this additional result, we hope to have further clarified that we propose a faster but equally performing strategy compared to other optimization schemes for learning acyclic graphs, without aiming to tackle the causal discovery problem. As requested by the reviewer, the manuscript now provides results on both "*more challenging settings*" and against the sortnregress baseline, in addition to an extensive comparison with other optimization schemes in their exact empirical scenario.

Reviewer comment (Experiments):
After discussion, the authors added the literature on order-based methods to their related work and reported a comparison with one of them (DAGuerreotype) in the appendix. However, this comparison is again carried out on trivial settings. Moreover the authors justified not thoroughly comparing with this line of works because they are supposedly not scalable. However one of the VP-DAG variants is one of the few methods with quadratic complexity (like the proposed method) so it is not clear how the authors can make this claim without empirical evidence.

Author response:
**Related Works.** Concerning the first major issue identified in the original review, we would like to remark that our related works section now references the reported works, explains why they were not included in the main empirical comparison, and finally backs this choice with dedicated experiments against several DAGuerreotype configurations on four distinct datasets.

Reviewer comment (Novelty):
The main weakness of the paper is its novelty. The re-parametrization itself is not new, that is, the idea of decomposing a weighted adjacency matrix by a real matrix and orientation matrix. Comparing to NOCurl, the change is basically from ReLU to Sigmoid. However, I should give credits to the authors for making this arguably small change work well in practice.

Author response:
**Novelty.** Considering the weakness identified by the reviewer, we acknowledge in the main body the similarities between ours and NOCURL parameterization. However, given the theoretical guarantees ensured by our smooth orientation matrix (Theorems 1 and 2), we are able to define an optimization scheme significantly different from NOCURL. As we comment in Appendix D.2, NOCURL proposes an algorithm that extracts the ordering from a solution obtained through NOTEARS, which in turn requires evaluating acyclicity during training. On the other hand, COSMO jointly learns the ordering, is entirely differentiable, and does not evaluate acyclicity at any optimization step.

Reviewer comment (Theory):
Consider a $p$ vector with all entries being different. Then, for some arbitrarily small $\epsilon$, it seems to me that the orientation matrix is the adjacency matrix of a fully-connected DAG. In some sense, $\epsilon$ controls the sparsity level of the output, however, this is problematic because $\epsilon$ is not something being optimized but instead fixed a-priori. Please correct me if I am wrong.

Author response:
**Epsilon.** The reviewer is correct that with a sufficient small $\varepsilon$, the smooth orientation matrix $S_{t,\varepsilon}$ would be a fully connected matrix. However the sparsity of the overall solution is controlled also by the directed matrix $\mathbf{H}$, since the overall DAG is given by the product $\mathbf{W}=\mathbf{H}*S_{t,\varepsilon}(\mathbf{p})$. Therefore, even if an orientation would be complete, we can still learn a sparse graph given the sparsity of $\mathbf{H}$. Notably, $\varepsilon$ cannot be zero to avoid cycles between variables with the same priority.

Reviewer comment (Experiments):
In the paragraph about "Direct Matrix Regularization". For the values found for $H$ and $p$ after optimizing, why not simply threshold the smooth orientation matrix by a small amount so that the "small entries" are set to zero and are not affected by the possibly large values of $H$. Given that linear models with equal variances are identifiable from the score function, L2 regularization of the weights should not be necessary and I wonder if doing this affects the performance of COSMO.

Author response:
**L2 Regularization.** We thank the reviewer for the suggestion on thresholding the smooth orientation matrix during training. We experimented with a similar approach during preliminary experiments, where we used a hard threshold to test the difference between priorities in the forward pass, while keeping the sigmoid for the backward pass. However, we found it beneficial to avoid thresholding in both passes. As a further reference, we included in Appendix F an experiment on how acyclicity evolves during training for different values of L2 regularization.

Reviewer comment (Reproducibility):
I think more details about the optimization procedure should go to the main text. In particular, the scheduling for $t$ seems very important and it was not clear how it was set in the main text.

Author response:
**Temperature and Acyclicity.** In our methodological section we report that we perform cosine annealing of the temperature during training, but we have added a pointer to the Appendix where the choice of temperature and other hyperparameters range is discussed. Further, the matrix is guaranteed to be acyclic *without* thresholding, which we only perform to remove spurious edges. We included in Appendix G a visualization of the optimization process that shows the trend of acyclicity and performance metrics against the temperature and highlights how COSMO reaches an acyclic solution even for non-zero temperatures.

Reviewer comment (Evaluation):
Related to my point above, I might have missed this since I only briefly looked at the appendix. But the NOTEARS $h$ is not a good proxy for acyclicity as discussed by Bello et al. (2022). In fact, given that your method does not rely on $h$, why not simply threshold the smooth orientation matrix by a very small amount and check in $O(|V|+|E|)$ that the adjacency corresponds to a DAG?

Author response:
**Temperature and Acyclicity.** In our methodological section we report that we perform cosine annealing of the temperature during training, but we have added a pointer to the Appendix where the choice of temperature and other hyperparameters range is discussed. Further, the matrix is guaranteed to be acyclic *without* thresholding, which we only perform to remove spurious edges. We included in Appendix G a visualization of the optimization process that shows the trend of acyclicity and performance metrics against the temperature and highlights how COSMO reaches an acyclic solution even for non-zero temperatures.

Reviewer comment (Reproducibility):
In the experiments, the runtime for DAGMA looks different to what was reported by Bello et al. (2022). These might be because of the available implementation of DAGMA is using different hyperparameters for the optimization, but I am curious what values were used in the comparison for the main text.

Author response:
**Further Additions.** We added in Appendix C.4 a table reporting the hyperparameter ranges for each baseline, including DAGMA. Also, we have updated the references according to yours and other reviewers comments, please refer to the extended discussion on related works in the updated draft.

Reviewer comment (Experiments):
I should also note a related work on learning DAGs by Deng et al. (2023) "Optimizing NOTEARS objectives via topological swaps", which obtains better performances in structure estimation although somewhat slower.

Author response:
**Further Additions.** We added in Appendix C.4 a table reporting the hyperparameter ranges for each baseline, including DAGMA. Also, we have updated the references according to yours and other reviewers comments, please refer to the extended discussion on related works in the updated draft.

Reviewer comment (Experiments):
Figure 3: since you showed the results with 1e4 nodes, what is the accuracy performance of these methods with large d?

Author response:
**Figure 3.** The goal of the experiment reported in Figure 3 is to test the average training epoch time of each method. To this end, we limited the training of each method to a small and finite and small number of epochs and computed the average duration. The results are therefore not meaningful in terms of accuracy or performance.

Reviewer comment (Theory):
Page 13 in appendix, last line: how did the denominator become $(1 + e^{-\epsilon})^n$ in the last step?

Author response:
**Proof of Theorem 3.** We thank the reviewer for the remark. There was indeed a typo which we have corrected in the rebuttal of the manuscript.

Reviewer comment (Novelty):
Authors seem to over claim the contributions and often make some imprecise statements. The formulation is based on the existing nocurl formulation, with $W = H \cdot f(p)$. Nocurl also learn a topological order with p along with the typical weighted adjacency matrix H. The difference is that nocurl learns topology from a preliminary solution, while the proposed COSMO optimizes them jointly with a smoothed formulation. This in itself is a worthy contribution, no need to make exaggeration statements such as claiming "the first unconstrained optimization approach" (which would be nocurl-u by authors' definition) and nocurl is "a slightly similar model". Authors should discuss the differences.

Author response:
**Contribution Claim.** In Appendix D.2 we discuss the differences between our parameterization and the one used by NOCURL. The section was already referenced in the related works, but thanks to the reviewer suggestion we also added a pointer from the methodological section. There, we also changed the claim in “similar model [...] with a significantly different optimization scheme” to better highlight the differences. Also, we have adjusted our claim on the primacy of unconstrained optimization models according to the reviewer’s opinion.

Reviewer comment (Evaluation):
Learning topology seems to be a hard task by itself. Gradient-based approach may not do too well on it, which explains the poorer results when the number of variable is smaller. What are authors' observation on the change in topology over time (by the orientation matrix)? Does it have significant change in the beginning and then barely change toward the end of optimization steps? How does the temperature annealing change the topology over time?

Author response:
**Topology Change.** Overall, we observed that the topology mostly changes at the beginning of the optimization process and at the end, when the temperature approaches zero. We also consistently notice that the model reaches acyclic solutions for non-zero temperatures. We back this claim by introducing Appendix G, where we report a visualization of the smooth orientation matrix, the directed matrix and the overall graph against the trends of temperature, dagness and NHD.  

> Is there an explanation for the changes at the end of optimization process?

At the end of the optimization process, as soon as the smooth orientation matrix $S_{t,\varepsilon}(\mathbf{p})$ turns acyclic, all the arcs non-respecting the partial order induced by the priority vector $\mathbf{p}$ are effectively pruned. Then, in the final optimization epochs, when the target adjacency matrix $\mathbf{W}$ is already acyclic but the temperature $t$ still non-zero, the direct acyclic matrix $\mathbf{H}$ might still have non-zero gradients and thus adjust or prune further arcs.

> Does it mean that the topology changes in the beginning and then changes again (maybe once) at the end?

Yes, even if the binary adjacencies vary across the whole optimization process, we consistently observe two main structural changes during the first and the last epochs, as highlighted by the trends in Figure 6 from Appendix G.

Reviewer comment (Presentation):
In Figure 1, are you learning a directed graph or a partially directed graph with undirected edges to be identified later? Please formally define the priority vector in the notation section.

Author response:
**Figure 1.** COSMO learns a directed acyclic graph as a function of a directed graph and a priority vector. The purpose of Figure 1 is to depict the parameterization that we also describe in intuitive terms in the Introduction section before summarizing our contributions. To clarify this point, in the rebuttal update of the manuscript, we remarked in the caption that the smooth acyclic orientation is a function of the priority vector and clarified the notation.

Reviewer comment (Theory):
Can you provide information on the computational time and sample complexity of COSMO?

Author response:
**Computational Time.** In terms of computational complexity, the construction of the directed acyclic graph given the priority vector $\mathbf{p}$ and the directed matrix $\mathbf{H}$ requires a quadratic number of steps in the number of nodes. Other than complexity results, we report in each table the computation time of each model on each dataset. In terms of sample complexity, we train all models on datasets containing 1000 realizations for each graph type, in close adherence with the NOTEARS experimental testbed.

Reviewer comment (Theory):
How do you guarantee the identifiability of the recovered graph, or do the results lead to some equivalence class?

Author response:
**Identifiability and Classic Methods.** As we report in the background, the identifiability of the true underlying graph requires several assumptions on the function class and the loss function. What we propose is a general approach to optimize acyclic graphs, which our empirical results show to be faster while comparable in optimization with other continuous structure learning methods. The minimization of the MSE without further assumptions does not ensure *per se* particular results on the identifiability, which require a continuous structure learner, like ours, to be embedded in a more general causal discovery framework. As suggested, we remarked on this aspect in the Related Works, to better guide the reader in the context compared to classic structure learning or causal discovery methods.

Reviewer comment (Novelty):
In the related work section, consider covering some classic DAG structure learning methods.

Author response:
**Identifiability and Classic Methods.** As we report in the background, the identifiability of the true underlying graph requires several assumptions on the function class and the loss function. What we propose is a general approach to optimize acyclic graphs, which our empirical results show to be faster while comparable in optimization with other continuous structure learning methods. The minimization of the MSE without further assumptions does not ensure *per se* particular results on the identifiability, which require a continuous structure learner, like ours, to be embedded in a more general causal discovery framework. As suggested, we remarked on this aspect in the Related Works, to better guide the reader in the context compared to classic structure learning or causal discovery methods.

Reviewer comment (Experiments):
The method relies on the temperature parameter for the sigmoid function. While annealing the temperature is shown to be effective, selecting an appropriate temperature schedule could be a challenging task, and the method's performance may be sensitive to the choice of temperature.

Author response:
**Temperature and Initialization.** We agree with the reviewer that the temperature and the initial conditions are indeed sensitive hyperparameters that might affect the method. In particular, we discuss the initialization strategy of the priority vector in Section 4. We introduced in Appendix G a visualization of the optimization process during temperature annealing, which highlights how COSMO converges to an acyclic solution before reaching zero temperature while continuing to optimize the acyclic matrix. Finally, concerning Figure 4, we gave priority to yours and other reviewers' comments and will consider moving it to the appendix according to the requested additions from the discussion.

Reviewer comment (Experiments):
The effectiveness of this approach might depend on the initial conditions. Finding suitable initializations for the priority vector and other parameters could impact convergence and the quality of the learned DAG.

Author response:
**Temperature and Initialization.** We agree with the reviewer that the temperature and the initial conditions are indeed sensitive hyperparameters that might affect the method. In particular, we discuss the initialization strategy of the priority vector in Section 4. We introduced in Appendix G a visualization of the optimization process during temperature annealing, which highlights how COSMO converges to an acyclic solution before reaching zero temperature while continuing to optimize the acyclic matrix. Finally, concerning Figure 4, we gave priority to yours and other reviewers' comments and will consider moving it to the appendix according to the requested additions from the discussion.

Reviewer comment (Presentation):
Figure 4 does not seem very necessary, and it's not very clear. Consider moving Figure 4 to the appendix and using the space to discuss the sensitivity analysis of your method's parameter choices.

Author response:
**Temperature and Initialization.** We agree with the reviewer that the temperature and the initial conditions are indeed sensitive hyperparameters that might affect the method. In particular, we discuss the initialization strategy of the priority vector in Section 4. We introduced in Appendix G a visualization of the optimization process during temperature annealing, which highlights how COSMO converges to an acyclic solution before reaching zero temperature while continuing to optimize the acyclic matrix. Finally, concerning Figure 4, we gave priority to yours and other reviewers' comments and will consider moving it to the appendix according to the requested additions from the discussion.