Paper: Analytic DAG Constraints for Differentiable DAG Learning (ICLR-2024)

Reviewer comment (Writing):
In the paper, "dataset scale" is an important concept but this has not been properly introduced. What is the dataset scale and what do you mean by "dataset scale is known"? Do you provide extra information during model training?

Author response:
The data scale means the variance of the noise variable. For known scale problem, unnormalized raw data is provided and for unknown scale problem, normalized data is provided. During model estimation no other information is provided except for the unnormalized/normalized data. From the unnormalized data, we can get further information from the scale of variables while from normalized data we cannot get such information.

Reviewer comment (Writing):
In the proposition, the \(-n\) is for \( (I-\tilde{B}) \) or \( tr(I-\tilde{B}) \)? This can be quite misleading. For the previous discussion, I assume it is for \( (I-\tilde{B}) \)?

Author response:
We have updated the equation to make it clear.

Reviewer comment (Theory):
For "Choosing DAG constraints for different scenarios," the arguments made in that section are unclear. For example, why with known data scale, can the objective provide a larger gradient? Also, if we have a large constraint, it will still create many local optima even with an informative objective function, right? So the correct argument should be to achieve an appropriate balance between objective and constraints?

Author response:
We have added a Section to discuss the non-convexity of the DAG constraints by using the Hessian of the DAG constraints. We roughly show that constraints with a finite spectral radius suffer more from non-convexity as they have larger $c_i$. However, they also have the benefits of less gradient vanishing as larger $c_i$ cause a larger norm of gradients. Thus for problems such as a problem with known scale, as the MSE loss provides enough information, the non-convexity may not be a serious issue and we can choose constraints with finite convergence radius regardless of its stronger non-convexity. Meanwhile, for a problem with an unknown scale, the MSE loss cannot provide enough guidance we will have to choose constraints with less non-convexity, i.e. constraints with infinite convergence radius.

Reviewer comment (Writing):
In the experiment section, what is \( \otimes \)? Is it the Kronecker product? Why do you use this instead of \( \odot \)?

Author response:
We have fixed the typos.

Reviewer comment (Experiments):
For experiments 4.1 and 4.2, why does the dimensionality differ by a lot? For 4.1, the dimensionality starts at 200, but in 4.2, the highest is 50. I also want to see the performance of PC with known true data scale.

Author response:
Algorithms such as PC and GES are not scalable, to compare with them we have to use small graphs. The performance of PC is almost the same as the performance on unknown data scale, and it is worse than differentiable DAG learning used in our paper. This is widely reported in previous work. We have applied PC and GES on small graphs (up to 100 nodes) with known scale, and the result is consistent with previously reported ones, where PC and GES perform worse than continuous structure learning approaches. We will add these results in the future version.

Reviewer comment (Presentation):
First, the paper is overflowing the 9 pages limit ?

Author response:
We are sorry for the problem. Due to limited time there was a technique problem, which results in a very large black area around Figure 2. This issue is now fixed.

Reviewer comment (Theory):
What is the effect of the multiplication introduced in Eq 14 to get positivity ? This looks a bit hacky and wasn't done in previous related works ?

Author response:
This is the standard approach in continuous DAG learning. This trick may have pros and cons, for more details we refer to Dennis Wei, Tian Gao, and Yue Yu. Dags with No Fears: A closer look at continuous optimization for learning
bayesian networks. Advances in Neural Information Processing Systems, 33:3895–3906, 2020, and Ignavier Ng, Sébastien Lachapelle, Nan Rosemary Ke, Simon Lacoste-Julien, and Kun Zhang. On the convergence of continuous constrained optimization for structure learning. In International Conference on Artificial
Intelligence and Statistics, pp. 8176–8198. PMLR, 2022.

Reviewer comment (Writing):
The clarity of the paper degrades after page 6. The final paragraph of the experimental section and the conclusions are rushed and need revision. Given that the rest of the paper is mostly clear and well written, this alone wouldn't be too much of a problem. However, the paper also exceeds the 9 pages, and there is potential for "unfair" comparison with other authors who spent time making sure to respect the 9 page limit and polish the entire submission.

Author response:
We are sorry for the problem. Due to limited time there was a technique problem, which results in a very large black area around Figure 2. This issue is now fixed. We will also carefully revise the experiment section to make it more clear.

Reviewer comment (Theory):
The weakest part of the work is the connection between the convergence radius of the series and the trade-off between vanishing gradient and non-convexity. The authors should elaborate more on this, especially regarding the gradient vanishing aspect.

Author response:
We have now added a section, discussing the relationship between Hessian of DAG constraints and the convergence radius. The Hessian of the DAG constraints is symmetric but not positive semi-definite. One common way to convexify the problem is to add a term $\eta \mathbb{I}$ to the Hessian (equivalently, add $\eta ||\tilde{\mathbf{B}}||_F^2$) to the objective. If $\eta$ is larger than the Frobenius norm of the Hessian, the problem is convexified. This means that we can use the Frobenius of the Hessian to roughly measure the non-convexity of the DAG constraints.

The Frobenius norm of the Hessian, at some particular point $\tilde{\mathbf{B}}$, only depends on coefficients $c_i$ of the analytic function. From Eq (33) in the updated paper, it clearly shows that larger $c_i$ causes a large Frobenius norm of the Hessian, which roughly shows that constraints with a finite spectral radius suffer more from non-convexity as they have larger $c_i$. However, they also have the benefits of less gradient vanishing as larger $c_i$ cause a larger norm of gradients. 

For a problem with a known scale, it is well-known that the MSE loss provides sufficient information for estimating the graph structure, and thus the non-convexity would not cause too much problem as the MSE loss can help to pull the problem out of local minimum. Meanwhile, for a problem with an unknown scale, due to the symmetric property of the Gaussian distribution, the MSE loss, along with the DAG constraints, may suffer a lot from non-convexity as the MSE loss.

Reviewer comment (Experiments):
The experiments only pertain to synthetic data and do not report any comparison with score-based non-relaxed/discrete methods, such as those in references [1, 2, 3].

Author response:
Our algorithm is compared with a discrete score-based discrete method GES on a dataset with an unknown scale. Currently, most score-based non-relaxed/discrete approaches are not so scalable, and they only work on graphs with at most 100-200 nodes, meanwhile in most of our experiments the graph size can go up to 2000. In our revised version, we will add an experiment comparing with those methods on small graphs.

Reviewer comment (Theory):
Equation 16: Shouldn't it be $B\circ B$ in the Frobenius norm?

Author response:
In Eq (16), $\mathbf{B}$ is the adjacency matrix so that it appears in the Frobenius norm. The DAG constraints only work for a positive weighted adjacency matrix, and the weight in $\mathbf{B}$ can be either positive or negative. Meanwhile, $\mathbf{B}$ is a DAG if and only if $\mathbf{B}\odot \mathbf{B}$ is a DAG. Based on this we apply DAG constraints on $\mathbf{B}\odot \mathbf{B}$.

Reviewer comment (Theory):
Is the term "invex" appropriate for describing the analytical constraints? Or maybe Fig. 1 is misleading, as invex functions have stationary point $\implies$ global minimum.

Author response:
If we only consider the DAG constraints, it is invex. In the overall framework we further have constraints on the $\ell_1$ norm of the adjacency matrix. In this case, the curve of the DAG constraints becomes similar to Figure 1 due to the norm constraints. We will clarify this in the revised version.