Paper: Neural structure learning with stochastic differential equations (ICLR-2024)

Reviewer comment (Theory):
The paper aims to sample the structure of G and obtain a sparse G. However, it falls short in providing a thorough discussion of the reasons behind the choice of priors for structure sampling. Additionally, there is no exploration of whether achieving sparsity in G leads to improved estimation performance, or if it primarily results in more interpretable results.

Author response:
The reason we need the sparsity prior is because in theory, a fully connected graph will always have the greatest capacity to fit data. The purpose of incorporating a sparsity term is to avoid false positive edges, i.e. spurious edges that may slightly improve the ELBO but are not present in the underlying system. In real-world systems (e.g. DREAM3), we found this helped to more accurately predict the ground-truth graph.

L1 sparsity priors (Lasso) are the most common type of prior used in structure learning, so we adopt this for our method. That said, there are other possibilities. For example, another prior sometimes used in DAG structure learning (Eggeling et al. 2019) assigns probability $p(G) \propto \prod_{d=1}^{D} \frac{1}{{D-1 \choose |pa(d)|}}$, where $pa(d)$ is the set of parents of the $d^{\textnormal{th}}$ variable. In theory we can use any graph prior that assigns positive probability to every graph. The L1 prior turns out to be particularly convenient (in combination with our variational graph posterior) as the KL-divergence in Eq. (14) is tractable.

(Eggeling et al. 2019) Ralf Eggeling, Jussi Viinikka, Aleksis Vuoksenmaa, Mikko Koivisto. On Structure Priors for Learning Bayesian Networks. AISTATS 2019.

Reviewer comment (Writing):
I think the assumptions (1, 2) should be more clearly stated (I had to comb through the text to find them)

Author response:
Agreed; we have updated the manuscript to place the assumptions in the main text.

Reviewer comment (Theory):
In specific homogenous drift and diffusion processes assumption may not be able to model some types of dynamics, some discussion on that would be interesting

Author response:
This is definitely an interesting point, which we now discuss in the conclusion of the revised manuscript.  In a nutshell, with homogeneous drift and diffusion functions, SCOTCH cannot model non-stationary dynamics. For example, systems with obvious changepoints (e.g. climate science models) will not have time-homogenous drift/diffusion functions. Unfortunately, our theoretical results no longer hold if time-dependent graphs are allowed.

Reviewer comment (Experiments):
What is the effect of the sparsity prior on the structure? Can a different type of sparsity be used?

Author response:
The reason we need the sparsity prior is because in theory, a fully connected graph will always have the greatest capacity to fit data. The purpose of incorporating a sparsity term is to avoid false positive edges, i.e. spurious edges that may slightly improve likelihood but are not present in the underlying system. In real-world systems (e.g. DREAM3), we found this helped to more accurately predict the ground-truth graph.

L1-based sparsity priors (Lasso, group Lasso, adaptive group Lasso) are the most common type of prior used in structure learning, so we adopt this for our method. That said, there are other possibilities. For example, another prior sometimes used in DAG structure learning (Eggeling et al. 2019) assigns probability $p(G) \propto \prod_{d=1}^{D} \frac{1}{{D-1 \choose |pa(d)|}}$, where $pa(d)$ is the set of parents of the $d^{\textnormal{th}}$ variable. In theory we can use any graph prior that assigns positive probability to every graph. The L1 prior turns out to be particularly convenient (in combination with our variational graph posterior) as the KL-divergence in Eq. (14) is tractable. 

(Eggeling et al. 2019) Ralf Eggeling, Jussi Viinikka, Aleksis Vuoksenmaa, Mikko Koivisto. On Structure Priors for Learning Bayesian Networks. AISTATS 2019.