DAGMA: Learning DAGs via M -matrices and a Log-Determinant Acyclicity Characterization
Abstract
The combinatorial problem of learning directed acyclic graphs (DAGs) from data was recently framed as a purely continuous optimization problem by leveraging a differentiable acyclicity characterization of DAGs based on the trace of a matrix exponential function.
Existing acyclicity characterizations are based on the idea that powers of an adjacency matrix contain information about walks and cycles.
In this work, we propose a new acyclicity characterization based on the log-determinant (log-det) function, which leverages the nilpotency property of DAGs.
To deal with the inherent asymmetries of a DAG, we relate the domain of our log-det characterization to the set of M -matrices , which is a key difference to the classical log-det function deﬁned over the cone of positive deﬁnite matrices.
Similar to acyclicity functions previously proposed, our characterization is also exact and differentiable.
However, when compared to existing characterizations, our log-det function: (1) Is better at detecting large cycles; (2) Has better-behaved gradients; and (3) Its runtime is in practice about an order of magnitude faster.
From the optimization side, we drop the typically used augmented Lagrangian scheme and propose DAGMA ( Directed Acyclic Graphs via M -matrices for Acyclicity ), a method that resembles the central path for barrier methods.
Each point in the central path of DAGMA is a solution to an unconstrained problem regularized by our log-det function, then we show that at the limit of the central path the solution is guaranteed to be a DAG.
Finally, we provide extensive experiments for linear and nonlinear SEMs and show that our approach can reach large speedups and smaller structural Hamming distances against state-of-the-art methods.
Code implementing the proposed method is open-source and publicly available at https ://github.com/kevinsbello/dagma.
Introduction
Structural equation models (SEMs) [44] are a standard modeling tool in several ﬁelds such as economics, social sciences, genetics, and causal inference, to name a few.
Under this framework, in its general form, the value of each variable in the model is assigned by a general nonlinear, nonparametric function that takes as input the values of other variables in the model, thus, every
SEM can be associated to a graphical model.
In particular, we will consider graphical models that are directed acyclic graphs (DAGs).
A long-standing and active research area deals with the problem of learning the graphical structure (DAG) given passively observed data (a.k.a.
causal discovery).
Computationally, this problem is well-known to be NP-hard in general [10, 12], mainly due to the combinatorial nature of the space of
DAGs.
In this work, we will follow a score-based approach, a popular learning framework where the goal is to ﬁnd a DAG that minimizes a given score [21, 11].
Recently, Zheng et al.
[58] proposed an exact smooth nonconvex characterization of acyclicity which opened the door to solving the originally 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
combinatorial problem via a set of tools that work in the fully continuous regime (e.g., gradient-based methods).
Let W ∈ R
d
×
d be a weighted adjacency matrix of a graph G of d nodes, and let W ◦ W denote the
Hadamard product.
The acyclicity function introduced by Zheng et al.
[58] is deﬁned as h expm ( W ) =
Tr( e W ◦ W ) − d , where Tr denotes the trace of a matrix, and it was shown that h expm ( W ) =
if and only if W corresponds to a DAG.
A follow-up work [56] proposed another acyclicity function, which can be computed slightly faster, deﬁned as h poly ( W ) =
Tr(( I + 1 d W ◦ W ) d ) − d , where
I is the identity matrix.
It was similarly shown by Yu et al.
[56] that h poly ( W )
=
if and only if
W corresponds to a DAG.
While seemingly different, both h expm and h poly are functions of the form
Tr( (cid:80) dk
=0 c p ( W ◦ W ) k ) − d for some c k > 0 , as noted by Wei et al.
To the best of our knowledge, all subsequent work [e.g., 59, 26, 60, 37, 38, 36, 25, 41, to name a few] that has built upon the idea of using a continuous acyclicity characterization have used either h expm or h poly , or some acyclicity characterization in the form of a trace of a sum of matrix powers.
The latter should come as no surprise, after all, a nonzero diagonal entry of the matrix power ( W ◦ W )
k reveals the existence of a closed walk of length k in W .
Contributions.
In this work, we propose a new acyclicity function that, as h expm and h poly , is both an exact and a smooth acyclicity characterization but that also possesses several advantages when compared to h expm and h poly .
Speciﬁcally, we make the following set of contributions:
We propose a novel acyclicity characterization based on the log-determinant (log-det) function (see Theorem 1 and Section 3).
In contrast to the classical log-det function deﬁned over the cone of positive deﬁnite matrices, we deﬁne the domain of our log-det function to be the set of M -matrices due to the inherent asymmetries of DAGs.
To our knowledge, we are the ﬁrst to connect the notion of M -matrices to acyclicity and structure learning for DAGs.
We provide a detailed study of the properties of our log-det characterization in Section 3.1.
First, we establish the similarities of our log-det function to other existing functions such as h expm and h poly .
Second, we formally argue why these functions can be regarded as acyclicity regularizers, similar in spirit to the classical (cid:96) 1 and (cid:96) 2 regularizers.
Third, we show that our log-det function is an invex function, i.e., all stationary points are global minimum, moreover, these stationary points correspond to DAGs.
In Section 3.2, we present three arguments as to why our log-det characterization could be preferred over other existing acyclicity functions.
Brieﬂy, our log-det function is better at detecting large cycles, has better behaved gradients, and can be computed in about an order of magnitude faster than h expm and h poly .
Motivated by the properties of our log-det function, in Section 4, we present DAGMA ( Directed Acyclic Graphs via M -matrices for Acyclicity ), a method that resembles the widely known central path approach for barrier methods [39].
We show that, at the limit of the central path, the solution is guaranteed to be a DAG.
In contrast to the commonly adopted augmented Lagrangian scheme (originally proposed in [58]) each point in the central path of DAGMA is a solution to an unconstrained problem regularized by our log-det function.
Finally, in Section 5 and Appendix C, we provide extensive experiments for linear and nonlinear SEMs under different score functions (both least squares and log-likelihood), where we show that DAGMA is capable of obtaining DAGs with better accuracy , i .e. , lower structural Hamming distance (SHD), in a much faster way than the state-of-the-art.
1.1 Related work
The vast majority of methods for learning DAGs can be categorized into two groups: constraint-based algorithms, which rely on conditional independence tests; and score-based algorithms, which focus on ﬁnding a DAG that minimizes a given score/loss function.
We brieﬂy mention classical constraintbased methods as we follow a score-based approach.
[50] developed the PC algorithm, a popular general method that learns the Markov equivalence class.
Other algorithms such as [52, 29] are based on local Markov boundary search.
Finally, hybrid approaches that combine constraint-based learning with score-based learning, such as [53, 15].
In the line of score-based methods, popular score functions include BDeu [21], BIC [31], and MDL [7].
Works that study linear Gaussian SEMs include [2, 3, 16, 17, 32, 45], and for linear non-Gaussian
SEMs [27, 49].
For nonlinear SEMs, we note works on additive models [9, 13, 54], additive noise models [23, 46, 35], generalized linear models [43, 42, 19], and general nonlinear SEMs [34, 18].
More closely related to our work is the line of work built on the nonconvex continuous framework of Zheng et al.
[58], such as, [59, 26, 60, 37, 36, 25, 41].
In contrast to our work, all of the aforementioned methods rely on the nonconvex acyclicity functions h expm or h poly , and with the exception of [37], all of these works also use the augmented Lagrangian scheme.
Finally, the NoCurl method [57] also departs from using h expm , although no other acyclicity constraint is proposed.
Two immediate distinctions can be made to our work.
First, we propose a novel acyclicity function based on the log-det function which we show to be prefereable to h expm and h poly .
Second, we drop the commonly adopted augmented Lagrangian scheme to solve the constrained problem and instead follow a central path approach to leverage the barrier property of our log-det function.
Remark 1.
To avoid confusion, we also note that in the GOLEM method of Ng et al.
[37] the score includes a log-determinant function of the form log | det( I − W ) | which stems from the Gaussian log-likelihood .
While this expression is zero if W corresponds to a DAG, it is not an exact acyclicity characterization (i.e. log | det( I − W ) | = 0 does not imply W is a DAG).
By contrast, the use of
M -matrices in our work is crucial to translating the log-det function into a valid acyclicity regularizer.
Moreover, it is not obvious how to extend GOLEM to arbitrary score functions, as their analysis is speciﬁc to the Gaussian likelihood function.
Notation and background
Notation.
We use [ d ] to denote the set of integers { 1 . . .
d } .
For a square matrix A , we use λ
i ( A ) to denote its i -th minimum eigenvalue, and use ρ ( A ) to denote its spectral radius.
Also, we use Tr( A ) , and det( A ) to denote the trace and determinant of A .
For matrices A, B , we let A ◦ B represent the element-wise or Hadamard product, moreover, the expression A ≥ B is entrywise, i.e., A i,j ≥ B i,j .
Then, we say that a matrix A is nonnegative whenever A ≥ 0 .
For a complex number a + bi , we let (cid:60) ( a + bi ) =
a denote its real part.
We use (cid:107)·(cid:107) p to denote the vector (cid:96) p -norm, and (cid:107)·(cid:107)
L p is the
L p -norm on functions.
Lastly, i → j and i (cid:32) j represent an edge from i to j and a directed walk from i to j , respectively.
Let X =
( X 1 , . . .
, X d ) be a d -dimensional random vector.
In its general form, a (nonparametric) structural equation model (SEM) consists of a set of equations of the form:
[Equation]
where each f j : R d +1 → R is a nonlinear nonparametric function, and Z j is an exogenous variable representing errors due to omitted factors.
We consider the Markovian model, which assumes that each Z j is an independent random variable.
Note that each f j depends only on a subset of X (i.e., the parents of X j ) and Z j ; nonetheless, to simplify notation we ensure that each f j is deﬁned on the same space.
Then f = ( f 1 , . . .
, f d ) induces a graphical structure, where we focus on directed acyclic graphs.
For any joint distribution over Z = ( Z 1 , . . .
, Z d ) , the functions f j deﬁne a joint distribution P ( X ) over the observed data, and a graph G ( f ) via the dependencies in each f j .
Then, our goal is to learn G ( f ) given n i.i.d.
samples from P ( X ) .
In score-based learning, given a data matrix X =
[ x 1 , . . .
, x d ] ∈ R n × d , we deﬁne a score function Q ( f ; X ) to measure the ‘quality’ of a candidate
SEM as follows: Q ( f ; X ) =
(cid:80) dj
=1 loss ( x j , f j ( X )) , where we adopt the convention that f j ( X ) ∈
R n .
Here loss can be any loss function such as least squares loss ( u , v ) =
1 n (cid:80) ni
=1 ( u i − v i ) 2 or the log-likelihood function, often augmented with a penalty such as BIC or (cid:96) 1 .
Given the score function
Q and a family of functions F , we seek to ﬁnd the f ∈ F that minimizes the score, i.e.,
[Equation]
Similar to [59], we consider that each f j lives in a Sobolev space of square-integrable functions whose derivatives are also square integrable.
Then, let ∂ k f j denote the partial derivative of f j w.r.t.
X k , it is easy to see that f j is independent of X k
if and only if (cid:107) ∂ k f j (cid:107) L 2 = 0 .
With this observation, we construct the matrix W ( f ) ∈ R d × d with entries [ W ( f )] i,j def = (cid:107) ∂
i f j (cid:107) L 2 , which precisely encodes the graphical structure amongst the variables X j .
That is G ( f ) ∈ DAGs ⇐⇒ W ( f ) ∈ DAGs , where W is interpreted as the usual weighted adjacency matrix.
In practice, f is replaced with a ﬂexible family of parametrized functions such as deep neural networks, so that problem (2) is ﬁnite dimensional.
Finally, note that model (1) includes several models as special cases, e.g., additive noise models, generalized linear models, additive models, polynomial regression, and index models.
Previous work has studied the identiﬁability of several of these models, e.g., [23, 49, 46, 43, 42, 27].
In the sequel, we assume that the model is chosen such that the graph G ( f ) is uniquely deﬁned from (2).
A new characterization of acyclicity via log-determinant and M -matrices
In this section, we present our acyclicity characterization and study its properties.
To declutter notation, in this section we simply write W instead of W ( f ) to denote the weighted adjacency matrix of a graph; however, it should be clear that W depends on functions f j as explained in the previous section.
We develop our characterization by ﬁrst noting that for any nonnegative weighted adjacency matrix W , we have that W ∈ DAGs if and only if W is a nilpotent matrix, or equivalently, all the eigenvalues of W are zero, i.e., λ i ( W ) = 0 , ∀ i ∈ [ d ] .
Then, for any W ∈ R d × d , we have the following obvious implications:
[Equation]
Implication (4) can be thought of as a relaxation of acyclicity in the sense that all DAGs satisfy (4), but not all W that satisfy (4) are DAGs.
For example, let s = 1 and W ◦ W = (cid:2) 2 0 0 2 (cid:3) , then it is clear that det( sI − W ◦ W )
= 1 and, thus, (4) is satisﬁed; however, clearly W is not a DAG.
Thus, two immediate questions arise: (i) Does there exist a domain for W such that (4) =
⇒ (3) ?
(ii)
If so, what is the description of such domain?
We answer (i) in the afﬁrmative, and answer (ii) by relating the domain of W to the set of M -matrices, which is deﬁned below.
Deﬁnition 1 ( M -matrix 1 ) .
An M -matrix is a matrix A ∈ R d
× d of the form A = sI − B , where B ≥and s > ρ ( B ) .
M -matrices were introduced by Ostrowski [40] and arise in a variety of areas including input-output analysis in economics, linear complementarity problems in operations research, ﬁnite difference methods for partial differential equations, and Markov chains in stochastic processes.
To the best of our knowledge, we are the ﬁrst to connect the notion of M -matrices to graphical model structure learning through an acyclicity characterization.
The following proposition is an immediate consequence of Deﬁnition 1:
Proposition 1 (Berman and Plemmons [5]) .
Let A ∈ R d × d be an M -matrix, then:
(i) (cid:60) ( λ i ( A )) > 0 , for all
i ∈
[ d ] .
(ii)
A − 1 exists and is nonnegative, i.e., A − 1 ≥ 0 .
In the above, item (i) states that the eigenvalues of an M -matrix lie in the open right-half plane.
Matrices which satisfy the latter property are also known as positive stable matrices.
We thus have that M -matrices are special cases of positive stable matrices.
It follows that the determinant of any
M -matrix is positive.
This fact will be used for deﬁning h s ldet ( W ) , our acyclicity characterization given in Theorem 1.
Finally, the nonnegativity of the inverse from item (ii) will be used to understand some properties of the gradient of h s ldet ( W ) .
We now deﬁne the domain over which (4) =
⇒ (3) (see Theorem 1).
For any s > 0 , deﬁne
[Equation]
i.e., W s is the set of real matrices whose entry-wise square given by W ◦ W have spectral radius less than s .
The following lemma lists some relevant properties of W s .
Lemma 1.
Let W s deﬁned as in (5) .
Then, for all s > 0 :
(i) DAGs ⊂
W s .
(ii) W s is path-connected.
(iii) W s ⊂ W t for any t > s .
In Lemma 1, item (i) implies that the W we look for is in the interior of W s ; item (ii) indicates that we can ﬁnd a path from any point in W s to any DAG without leaving the set W s ; item (iii) shows that one can vary s to enlarge or shrink the set W s .
Having deﬁned the domain set W s , we now deﬁne our acyclicity characterization h s ldet : W s → R .
Recall that item (i) in Proposition 1 implies that applying the logarithm function to the determinant of an M -matrix is always well deﬁned, which motivates our following result.
Theorem 1 (Log-determinant characterization) .
Let s > 0 and let h s ldet : W s → R be deﬁned as h s ldet ( W ) def = − log det( sI − W ◦ W ) + d log s .
Then, the following holds:
(i) h s ldet ( W ) ≥ 0 , with h s ldet ( W ) =
if and only if W is a DAG.
(ii) ∇ h s ldet ( W ) = 2( sI − W ◦ W ) −(cid:62) ◦ W , with ∇ h s ldet ( W )
=
if and only if W is a DAG.
3.1 Properties of h
s ldet ( W )
In this section, we list several properties of our acyclicity characterization.
The ﬁrst property we discuss is related to the entries of ∇ h s ldet ( W ) .
Lemma 2.
For all i, j ∈ [ d ] , [ ∇ h s ldet ( W )] i,j = 0 if and only if W i,j = 0 or there is no directed walk from j to i .
Equivalently, [ ∇ h s ldet ( W )] i,j (cid:54)
=
if and only if the edge i → j is part of some cycle in
W .
Finally, whenever [ ∇ h s ldet ( W )] i,j (cid:54) = 0 , we have that sign([ ∇ h s ldet ( W )] i,j )
= sign( W i,j ) .
The lemma above characterizes the nonzero entries and their signs of the gradient of h s ldet .
This property formally offers a regularizer perspective for h s ldet , which we highlight next.
Remark 2 (A regularizer viewpoint) .
The function h s ldet promotes small parameters values in much the same way that the classical (cid:96) 1 and (cid:96) 2 regularizers do.
In contrast to the latter regularizers, h s ldet will only shrink the value of a parameter W i,j if and only if the edge ( i, j ) is part of some cycle in W , as prescribed by Lemma 2.
Recall that h expm ( W ) =
Tr( e W ◦ W ) − d
and h poly ( W ) =
Tr(( I + 1 d W ◦ W ) d ) − d .
It was noted by [55] that acyclicity characterizations of the form
Tr( (cid:80) dp
=1 c p ( W ◦ W ) d ) for c p > 0 also have the property in Lemma 2.
This implies that h expm and h poly hold the property above and can also be interpreted as acyclicity regularizers.
We note that the interesting part here is that h s ldet holds this property besides being different in nature to h expm and h poly .
Next, we state an important consequence of Lemma 2, which is related to the direction of ∇ h s ldet ( W ) .
Corollary 1.
At any W ∈ W s , the negative gradient ∇ h s ldet ( W ) points towards the interior of W s .
In optimization, the Hessian matrix plays an important role as it contains relevant information about saddle points and local extrema of a function, and is key to Newton-type methods.
Another appealing property of h s ldet ( W ) is that it has a Hessian described by a simple closed-form expression.
Lemma 3.
The Hessian of h s ldet ( W ) , which resides in R d 2 × d 2 , is given by:
[Equation]
where N = ( sI − W ◦ W ) − 1 , ⊗ denotes the Kronecker product, and K dd is the d 2 × d 2 commutation matrix such that K dd vec( A ) =
vec( A (cid:62) ) , for any d × d matrix A .
Here we note that among h expm , h poly and h s ldet , only h s ldet has a tractable expression for the Hessian.
Furthermore, note that ∇ 2 h s ldet ( W ) is indexed by vertex pairs so that the entry [ ∇ 2 h s ldet ( W )]
( k,l ) , ( p,q ) corresponds to the second partial derivative
[Equation]
Using Lemma 3, we can characterize the nonzero entries, and their signs, of the Hessian of h s ldet .
Corollary 2.
The entries of the Hessian ∇ 2 h s ldet ( W ) are described as follows:
[Equation]
, where N = ( sI − W ◦ W ) − 1 .
Moreover, an off-diagonal entry [ ∇ 2 h s ldet ( W )] ( k,l ) , ( p,q ) is nonzero if and only if there exists a cycle in W of the form q → p (cid:32) l → k (cid:32) q , and has a sign equal to sign( W l,k W q,p ) .
Lastly, a diagonal entry [ ∇ 2 h s ldet ( W )] ( k,l ) , ( k,l ) is nonzero if and only if there exists a directed walk from k to l , and its sign is always positive.
Recall from Theorem 1 that all DAGs attain the minimum value and are critical points of h s ldet ; thus,
DAGs are local (and global) minimum of h s ldet and the Hessian matrix evaluated at a DAG must be positive (semi)deﬁnite.
Let us corroborate the latter, when W is a DAG, from Corollary 2 we have that the off-diagonal elements of ∇ 2 h s ldet ( W ) are zero, while the diagonal entries are nonnegative.
That is, the Hessian is positive semideﬁnite whenever W is a DAG.
This implies that all its stationary points are global minima: Such functions are called invex [20, 30].
Corollary 3.
Let s > 0 .
Then, h s ldet ( W ) is an invex function, i.e., all its stationary points are global minima, and these correspond to DAGs.
We note that even though h expm and h poly are also invex functions, they were not explicitly considered as invex functions before.
In fact, it was noted in [58] that DAGs were global minima of h expm but no characterization of its stationary points were given.
Wei et al.
[55] noted that DAGs were stationary points of h expm and h poly but the notion of invexity was not explicitly stated.
Remark 3 (A dynamical system perspective) .
The importance of invexity here is that in the eyes of h s ldet , all DAGs are the same.
That is, DAGs correspond to the set of attractors in h s ldet , and depending on the initial condition, the system will converge to a different attractor.
This offers the following viewpoint for the role of the score function Q ( f ; X ) , namely, “use the score Q to ﬁnd a basin of attraction such that the force ﬁeld of h s ldet will dictate the trajectory towards a DAG that is equal or close to the ground-truth”.
In Figure 1, we illustrate in a toy example the properties discussed in this subsection.
[Figure]
Why the log-determinant regularizer is preferable to existing acyclicity regularizers
In this section, we present three arguments as to why one should use h s ldet instead of existing functions such as h expm and h poly .
We invite the reader to look at Appendix B for additional details.
Argument (i).
h s ldet does not diminish cycles of any length.
Let us expand the functions h expm and h poly in their sum of matrix powers form, that is, h expm ( W ) =
(cid:80) ∞
k =0 1 / k !
Tr(( W ◦ W ) k ) − d and h poly ( W ) =
(cid:80) [-d k =
-]dk =0 ( dk ) /
d k Tr(( W ◦ W ) k ) − d .
Recall also that the entry [( W ◦ W )
k
i,i represents the sum of weighted walks from node i to node i of length k , where each edge has weight w 2 u,v .
Thus, one can notice that if W has cycles of length k , their contribution to h expm and h poly are diminished by 1 / k !
and ( dk ) /
d k , respectively.
Numerically, the latter can be problematic for the following reason:
Cycles of length k can go undetected even for small values of d and k .
In practice, a value of h expm , h poly ∈ [10 − 8 , 10 − 5 ] is typically regarded as zero [55, 58].
Consider a cycle graph of d nodes where each edge weight is +1 or − 1 .
The plot in Figure 2 shows how the values of h expm and h poly decay much faster than that of h s ldet ; in fact, at d = 13 we already observe h expm ( W ) ≈ 10 − 9 and h poly ( W ) ≈ 10 − 14 , i.e., cycles of length at least 13 would be numerically undetected by h expm and h poly .
In contrast, we observe that the value of the log-det function remains bounded away from zero and is able to detect larger cycles.
[Figure]
In contrast, letting s = 1 . 001 , we observe that h s ldet can stay away from zero even for a cycle graph of 100 nodes.
Finally, we note a similar pattern for the entrywise
(cid:96) ∞ norm of the gradients.
At ﬁrst glance, it might seem difﬁcult to directly compare the value of h s ldet to h expm and h poly .
We next show that when s = 1 , h s =1 ldet is an upper bound to h expm and h poly .
Lemma 4.
For all W ∈ W s =1 , we have h poly ( W ) ≤ h expm ( W ) ≤ h s =1 ldet ( W ) .
The lemma above shows that in spite of h s =1 ldet , h expm , and h poly being exact acyclicity characterizations, h s =1 ldet will attain the largest value.
Argument (ii).
h s ldet has better behaved gradients.
Similar to argument (i), we show in Appendix
B that h expm and h poly are susceptible to vanishing gradients even when the graph contains cycles (see Figure 2).
The following lemma states that the magnitude of each entry of ∇ h s ldet at least as large as the magnitude of the corresponding entry of ∇ h expm and ∇ h poly , and hence, h ldet has larger gradients to guide optimization.
Lemma 5.
For any walk of length k , its contribution to the gradients ∇ h expm ( W ) and ∇ h poly ( W ) are diminished by 1 / k !
and ( d − 1 k ) / ( d − 1) k , respectively.
In contrast, ∇ h s =1 ldet ( W ) does not diminish any walk of any length.
This implies that |∇ h poly ( W ) | ≤ |∇ h expm ( W ) | ≤ |∇ h s =1 ldet ( W ) | .
Argument (iii).
Computing h s ldet and ∇ h s ldet is empirically faster.
Even though h s ldet , h expm , and h poly all three share the same computational complexity of O
( d 3 ) , in practice h s ldet can be computed in about an order of magnitude faster than h expm and h poly .
In Figure 3, we compare the runtimes of h s ldet , h expm and h poly for randomly generated matrices, where we observe that computing h s ldet can be 10x faster than h expm and h poly .
See Appendix B for further details.
Optimization
In the previous section we argued why the log-det function should be preferred in practice.
Let f θ denote a model with parameters θ for the functions f j in (1), e.g., neural networks as in [59].
In this section we turn to the problem of minimizing a given score function Q ( f θ ; X ) constrained to h s ldet ( W ( θ ))
= 0 .
That is, we aim to solve:
[Equation]
where we include the (cid:96) 1 regularizer to promote sparse solutions. llllll l l l l l l l l l llllll l l l l l l l l l llllll l l l l l l l l l llllll l l l l l l l l l llllll l l l l l l l l l llllll l l l l l l l l l
[Figure]
= 1 ( W ) [Figure] W )
[Figure] Since the inception of the purely continuous framework for learning DAGs in [58], almost all followup work uses the augmented Lagrangian (AML) scheme to tackle problem (6), and L-BFGS-B [39] for solving the sequence of unconstrained problems.
Motivated by the properties of h s ldet given in 3.1, we propose a simpler scheme named DAGMA, based on solving a sequence of unconstrained problems in which h s ldet is simply seen as a regularizer.
DAGMA resembles the central path approach of barrier methods [8, 39], or the classical path-following approach for solving lasso problems [e.g.
Our method is given in Algorithm 1.
Algorithm
DAGMA
solve θ ( t +1) =
arg min θ µ ( t ) ( Q ( f θ ; X ) + β 1 (cid:107) θ (cid:107) 1 ) + h s ldet ( W ( θ )) 4:
Set µ ( t +1) =
αµ ( t )
Ensure: W ( θ ( T ) )
The following lemma states that DAGMA will return a DAG at the limit of the central path.
This is a critical distinction against existing methods, many of which rely on some type of post-processing (e.g. thresholding) to ensure that the solution is a DAG.
Lemma 6.
Algorithm 1 is guaranteed to return a DAG whenever µ ( t ) → 0 .
4.1 Practical Considerations
As in barrier methods, where it is required to start at the interior of the feasibility region, in Algorithm 1, we require that the initial point W ( θ (0) ) be inside W s .
This is very easy to achieve since the zero matrix is in the interior of W s for any s > 0 ; therefore, in our experiments we simply set θ (0) = 0 .
Note that in Algorithm 1, we let µ ( t ) decrease by a constant factor at each iteration; however, it is possible to specify explicitly the value of each µ ( t ) , e g., for T = 4 , we can let µ = { 1 , 0 . 1 , 0 .
Regarding the choice of s , in principle s could take any value greater than zero since DAGs are inside W s for any s > 0 .
Similar to µ , it is also possible to let s vary at each iteration, e.g., for T = 4 , we can set s = { 1 , 0 .
In practive, we observe that slightly decreasing s can help to obtain larger gradients as W gets closer to a DAG.
Note, however, that letting s be equal or close to 1 is generally easier to optimize than setting s closer to zero, the reason being that for smaller values of s the volume of W s is smaller and will require much smaller learning rates to stay inside W s , hence affecting convergence.
Finally, we do not specify how to solve line 3 in Algorithm 1, this is because we leave the door open for different solvers to be used.
For our experiments in the next section, we solve line 3 by using a ﬁrst-order method with the ADAM optimizer [24], which works remarkably well as shown in our experiments.
It remains as future work to exploit the Hessian structure of h ldet given in Lemma 3 for second-order methods.
Experiments
We compare our method against GES [11], PC [51], NOTEARS [58], and GOLEM [37] on both linear and nonlinear SEMs.
In Appendix C, we specify which existing implementation we used for each of the aforementioned methods.
Consistent with previous work in this area (e.g., NOTEARS and follow up work), we have not performed any hyperparameter optimization: This is to avoid presenting unintentionally biased results.
As a concrete example, for each of the SEM settings, we simply chose a reasonable value for the (cid:96) 1 penalty coefﬁcient and used that same value for all graphs across many different numbers of nodes.
Our experimental setting is similar to [58, 59].
For the main text, we present only a small fraction of all our experiments. Moreover, since the accuracy of certain methods were signiﬁcantly lower than other methods, we report results only against the most competitive ones; full results for all settings and methods can be found in Appendix C. Linear Models.
In Appendix C.1, we report results for linear SEMs with Gaussian, Gumbel, and exponential noises, and use the least squares loss.
For small to moderate number of nodes, see
Appendix C.1.1; for large number of nodes, see Appendix C.1.2; for denser graphs, see Appendix C.1.4; and for a comparison against GOLEM for sparser graphs, see Appendix C.1.3. Nonlinear Models. In Appendix C.2, we report results for nonlinear SEMs with binary and continuous data.
For binary data, we use a logistic model for each structural equation, and use the log-likelihood loss as the score, we report results for small to large number of nodes in Appendix
C.2.1.
For continuous data, we consider the continuous additive noise model with Gaussian noise [9], where each nonlinear relationship is modeled by a multilayer perceptron, and use the log-likelihood loss as the score, we report results for small to moderate number of nodes in Appendix C.2.2.
In the following ﬁgures, ER4 and SF4 denote Erd˝os-Rényi and scale-free graphs, respectively, where for each number of nodes d , each graph has in expectation 4 d edges.
It is worth noting that the empirical settings by Zheng et al.
[58, 59] consider graph models such as ER1, ER2, SF1, and SF2.
Here we focus on the hardest setting, i.e., ER4 and SF4 graphs.
For linear SEMs, Figure 4 shows results for graphs with d ∈ [20 , 100] , and Figure 5 shows results for graphs with d ∈ [200 , 1000] .
In both regimes, we note that DAGMA obtains signiﬁcant speedups and improvements in terms of structural accuracy (SHD) against NOTEARS and GOLEM, even though GOLEM is speciﬁc to and specialized for linear Gaussian SEMs.
For nonlinear SEMs, Figure 6 shows results for logistic models, we similarly observe that DAGMA attains major speedups and improvements on SHD against NOTEARS [58].
Finally, for nonlinear models using neural networks, we observe that DAGMA is comparable in SHD to the NONLINEAR NOTEARS [59] but obtains signiﬁcant speedups.
Again, we invite the reader to look at Appendix C for more details and additional experiments.
Final Remarks
A relevant assumption in this work is that of sufﬁciency, that is, there are no hidden variables that are a common cause of at least two observed variables.
While this assumption is widely used for structure learning, we nonetheless highlight that in practice it is very difﬁcult to ﬁnd scenarios where such assumption holds.
As with all work that assumes sufﬁciency, our work is an important necessary step to understanding settings with hidden variables.
Finally, we note that the work by [6] proposes a differentiable approach for ADMG for the semi-Markovian case using h expm .
It is left for future work to explore the performance of such method using h ldet .
Another important limitation of this and previous work on the continuous framework for learning
DAGs is that of providing guarantees on the learned structure.
As in real-life applications one does not have access to the ground-truth DAG, there is much uncertainty as to whether an edge in the predicted
DAG actually corresponds to a causal relation.
Thus, there is still a need for formal guarantees under the continuous framework.
Acknowledgments and Disclosure of Funding
[Figure] [Figure] l ll ll l l l l l l ll ll l l l l l Logistic E R
[Figure] t r u c t u r a l
H a mm i
[Figure] D i s t an c e [Figure]
l [Figure] l l l [Figure]
l
[Figure]
u c
[Figure]
u
[Figure]
l
[Figure]
i s t
[Figure] c e l l l l l l l l [Figure]
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (See Section 6) (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (See Section D) (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (See Sectionand 3) (b) Did you include complete proofs of all theoretical results?
[Yes] (See Section A) 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (
See Section
C)
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (See Section C) (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (See Section C)
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] (See Section C) 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (See Section C) (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]