DAGMA: Learning DAGs via M -matrices and a Log-Determinant Acyclicity Characterization
Abstract
The combinatorial problem of learning directed acyclic graphs (DAGs) from data was recently framed as a purely continuous optimization problem by leveraging a differentiable acyclicity characterization of DAGs based on the trace of a matrix exponential function.
Existing acyclicity characterizations are based on the idea that powers of an adjacency matrix contain information about walks and cycles.
In this work, we propose a new acyclicity characterization based on the log-determinant (log-det) function, which leverages the nilpotency property of DAGs.
To deal with the inherent asymmetries of a DAG, we relate the domain of our log-det characterization to the set of M -matrices , which is a key difference to the classical log-det function deﬁned over the cone of positive deﬁnite matrices.
Similar to acyclicity functions previously proposed, our characterization is also exact and differentiable.
However, when compared to existing characterizations, our log-det function: (1) Is better at detecting large cycles; (2) Has better-behaved gradients; and (3) Its runtime is in practice about an order of magnitude faster.
From the optimization side, we drop the typically used augmented Lagrangian scheme and propose DAGMA ( Directed Acyclic Graphs via M -matrices for Acyclicity ), a method that resembles the central path for barrier methods.
Each point in the central path of DAGMA is a solution to an unconstrained problem regularized by our log-det function, then we show that at the limit of the central path the solution is guaranteed to be a DAG.
Finally, we provide extensive experiments for linear and nonlinear SEMs and show that our approach can reach large speed-ups and smaller structural Hamming distances against state-of-the-art methods.
Introduction
Structural equation models (SEMs) [43] are a standard modeling tool in several ﬁelds such as economics, social sciences, genetics, and causal inference, to name a few.
Under this framework, in its general form, the value of each variable in the model is assigned by a general nonlinear, nonparametric function that takes as input the values of other variables in the model, thus, every
SEM can be associated to a graphical model.
In particular, we will consider graphical models that are directed acyclic graphs (DAGs).
A long-standing and active research area deals with the problem of learning the graphical structure (DAG) given passively observed data (a.k.a.
causal discovery).
Computationally, this problem is well-known to be NP-hard in general [10, 12], mainly due to the combinatorial nature of the space of
DAGs.
In this work, we will follow a score-based approach, a popular learning framework where the goal is to ﬁnd a DAG that minimizes a given score [21, 11].
Recently, Zheng et al.
[57] proposed an exact smooth nonconvex characterization of acyclicity which opened the door to solving the originally combinatorial problem via a set of tools that work in the fully continuous regime (e.g., gradient-based methods).
Let W ∈ R
d
×
d be a weighted adjacency matrix of a graph G of d nodes, and let W ◦ W denote the
Hadamard product.
The acyclicity function introduced by Zheng et al.
[57] is deﬁned as h expm ( W ) =
Tr( e W ◦ W ) − d , where Tr denotes the trace of a matrix, and it was shown that h expm ( W ) =
if and only if W corresponds to a DAG.
A follow-up work [55] proposed another acyclicity function, which can be computed slightly faster, deﬁned as h poly ( W ) =
Tr(( I + 1 d W ◦ W ) d ) − d , where
I is the identity matrix.
It was similarly shown by Yu et al.
[55] that
h poly ( W ) =
if and only if
W corresponds to a DAG.
While seemingly different, both h expm and h poly are functions of the form
Tr( (cid:80) dk
=0 c p ( W ◦ W ) k ) − d for some c k > 0 , as noted by Wei et al.
To the best of our knowledge, all subsequent work [e.g., 58, 26, 59, 37, 36, 25, 40, to name a few] that has built upon the idea of using a continuous acyclicity characterization have used either h expm or h poly , or some acyclicity characterization in the form of a trace of a sum of matrix powers.
The latter should come as no surprise, after all, a nonzero diagonal entry of the matrix power ( W ◦ W )
k reveals the existence of a closed walk of length k in W .
Contributions.
In this work, we propose a new acyclicity function that, as h expm and h poly , is both an exact and a smooth acyclicity characterization but that also possesses several advantages when compared to h expm and h poly .
Speciﬁcally, we make the following set of contributions:
We propose a novel acyclicity characterization based on the log-determinant (log-det) function (see Theorem 1 and Section 3).
In contrast to the classical log-det function deﬁned over the cone of positive deﬁnite matrices, we deﬁne the domain of our log-det function to be the set of M -matrices due to the inherent asymmetries of DAGs.
To our knowledge, we are the ﬁrst to connect the notion of M -matrices to acyclicity and structure learning for DAGs.
We provide a detailed study of the properties of our log-det characterization in Section 3.1.
First, we establish the similarities of our log-det function to other existing functions such as h expm and h poly .
Second, we formally argue why these functions can be regarded as acyclicity regularizers, similar in spirit to the classical (cid:96) 1 and (cid:96) 2 regularizers.
Third, we show that our log-det function is an invex function,
i
e., all stationary points are global minimum, moreover, these stationary points correspond to DAGs.
In Section 3.2, we present three arguments as to why our log-det characterization could be preferred over other existing acyclicity functions.
Brieﬂy, our log-det function is better at detecting large cycles, has better behaved gradients, and can be computed in about an order of magnitude faster than h expm and h poly .
Motivated by the properties of our log-det function, in Section 4, we present
DAGMA (
Directed
Acyclic
Graphs via
M -matrices for
Acyclicity
, a method that resembles
the
widely
known
central path approach for barrier methods
We show that, at the limit of the central path, the solution is guaranteed to be a DAG.
In contrast to the commonly adopted augmented Lagrangian
scheme (originally proposed in [57])
each point in
the
central path
of
DAGMA is a solution to an unconstrained
problem regularized
by our
log-det function.
Finally, in
Section 5 and Appendix C
, we provide
extensive
experiments
for linear and nonlinear SEMs
, where we show that
DAGMA is
capable
of obtaining DAGs with better accuracy i.
e., lower
structural Hamming
distance
(SHD)
, in a much faster way than the state-of-the-art.
1.1 Related work
The vast majority of methods for learning DAGs can be categorized into two groups: constraint-based algorithms, which rely on conditional independence tests; and score-based algorithms, which focus on ﬁnding a DAG that minimizes a given score/loss function.
We brieﬂy mention classical constraintbased methods as we follow a score-based approach.
[49] developed the PC algorithm, a popular general method that learns the Markov equivalence class.
Other algorithms such as [51, 29] are based on local Markov boundary search.
Finally, hybrid approaches that combine constraint-based learning with score-based learning, such as [52, 15].
In the line of score-based methods, popular score functions include BDeu [21], BIC [31], and MDL [7].
Works that study linear Gaussian SEMs include [2, 3, 16, 17, 32, 44], and for linear non-Gaussian
SEMs [27, 48].
For nonlinear SEMs, we note works on additive models [9, 13, 53], additive noise models [23, 45, 35], generalized linear models [42, 41, 19], and general nonlinear SEMs [34, 18].
More closely related to our work is the line of work built on the nonconvex continuous framework of Zheng et al.
[57], such as, [58, 26, 59, 37, 36, 25, 40].
In contrast to our work, all of the aforementioned methods rely on the nonconvex acyclicity functions h expm or h poly , and with the exception of [37], all of these works also use the augmented Lagrangian scheme.
Finally, the NoCurl method [56] also departs from using h expm , although no other acyclicity constraint is proposed.
Two immediate distinctions can be made to our work.
First, we propose a novel acyclicity function based on the log-det function which we show to be prefereable to h expm and h poly .
Second, we drop the commonly adopted augmented Lagrangian scheme to solve the constrained problem and instead follow a central path approach to leverage the barrier property of our log-det function.
Remark 1.
To avoid confusion, we also note that in the GOLEM method of Ng et al.
[37] the score includes a log-determinant function of the form log | det( I − W ) | which stems from the Gaussian log-likelihood .
While this expression is zero if W corresponds to a DAG, it is not an exact acyclicity characterization (i.e. log | det( I − W ) | = 0 does not imply W is a DAG).
By contrast, the use of
M -matrices in our work is crucial to translating the log-det function into a valid acyclicity regularizer.
Moreover, it is not obvious how to extend GOLEM to arbitrary score functions, as their analysis is speciﬁc to the Gaussian likelihood function.
Notation and background
Notation.
We use [ d ] to denote the set of integers { 1 . . .
d } .
For a square matrix A , we use λ
i ( A ) to denote its i -th minimum eigenvalue, and use ρ ( A ) to denote its spectral radius.
Also, we use Tr( A ) , and det( A ) to denote the trace and determinant of A .
For matrices A, B , we let A ◦ B represent the element-wise or Hadamard product, moreover, the expression A ≥ B is entrywise, i.e., A i,j ≥ B i,j .
Then, we say that a matrix A is nonnegative whenever A ≥ 0 .
For a complex number a + bi , we let (cid:60) ( a + bi ) =
a denote its real part.
We use (cid:107)·(cid:107) p to denote the vector (cid:96) p -norm, and (cid:107)·(cid:107)
L p is the
L p -norm on functions.
Lastly, i → j and i (cid:32) j represent an edge from i to j and a directed walk from i to j , respectively.
Let X =
( X 1 , . . .
, X d ) be a d -dimensional random vector.
In its general form, a (nonparametric) structural equation model (SEM) consists of a set of equations of the form:
[Equation]
where each f j : R d +1 → R is a nonlinear nonparametric function, and Z j is an exogenous variable representing errors due to omitted factors.
We consider the Markovian model, which assumes that each Z j is an independent random variable.
Note that each f j depends only on a subset of X (i.e., the parents of X j ) and Z j ; nonetheless, to simplify notation we ensure that each f j is deﬁned on the same space.
Then f = ( f 1 , . . .
, f d ) induces a graphical structure, where we focus on directed acyclic graphs.
For any joint distribution over Z = ( Z 1 , . . .
, Z d ) , the functions f j deﬁne a joint distribution P ( X ) over the observed data, and a graph G ( f ) via the dependencies in each f j .
Then, our goal is to learn G ( f ) given n i.i.d.
samples from P ( X ) .
In score-based learning, given a data matrix X =
[ x 1 , . . .
, x d ] ∈ R n × d , we deﬁne a score function Q ( f ; X ) to measure the ‘quality’ of a candidate
SEM as follows: Q ( f ; X ) =
(cid:80) dj
=1 loss ( x j , f j ( X )) , where we adopt the convention that f j ( X ) ∈
R n .
Here loss can be any loss function such as least squares loss ( u , v ) =
1 n (cid:80) ni
=1 ( u i − v i ) 2 or the log-likelihood function, often augmented with a penalty such as BIC or (cid:96) 1 .
Given the score function
Q and a family of functions F , we seek to ﬁnd the f ∈ F that minimizes the score, i.e.,
[Equation]
subject to
[Equation]
Similar to [58], we consider that each f j lives in a Sobolev space of square-integrable functions whose derivatives are also square integrable.
Then, let ∂ k f j denote the partial derivative of f j w.r.t.
X k , it is easy to see that f j is independent of X k
if and only if (cid:107) ∂ k f j (cid:107) L 2 = 0 .
With this observation, we construct the matrix W ( f ) ∈ R d × d with entries [ W ( f )] i,j def = (cid:107) ∂
i f j (cid:107) L 2 , which precisely encodes the graphical structure amongst the variables X j .
That is G ( f ) ∈ DAGs ⇐⇒ W ( f ) ∈ DAGs , where W is interpreted as the usual weighted adjacency matrix.
In practice, f is replaced with a ﬂexible family of parametrized functions such as deep neural networks, so that problem (2) is ﬁnite dimensional.
Finally, note that model (1) includes several models as special cases, e.g., additive noise models, generalized linear models, additive models, polynomial regression, and index models.
Previous work has studied the identiﬁability of several of these models, e.g., [23, 48, 45, 42, 41, 27].
In the sequel, we assume that the model is chosen such that the graph G ( f ) is uniquely deﬁned from (2).
A new characterization of acyclicity via log-determinant and M -matrices
In this section, we present our acyclicity characterization and study its properties.
To declutter notation, in this section we simply write W instead of W ( f ) to denote the weighted adjacency matrix of a graph; however, it should be clear that W depends on functions f j as explained in the previous section.
We develop our characterization by ﬁrst noting that for any weighted adjacency matrix W , we have that W ∈ DAGs if and only if W is a nilpotent matrix, or equivalently, all the eigenvalues of W are zero, i.e., λ i ( W ) = 0 , ∀ i ∈
[ d ] .
We then have the following obvious implications:
[Equation]
Implication (4) can be thought of as a relaxation of acyclicity in the sense that all DAGs satisfy (4), but not all W that satisfy (4) are DAGs.
For example, let s = 1 and W ◦ W = (cid:2)
[Equation]
2 (cid:3) , then it is clear that det( sI − W ◦ W ) =
1 and, thus, (4) is satisﬁed; however, clearly W is not a DAG.
Thus, two immediate questions arise: (i) Does there exist a domain for W such that (4) =
⇒ (3) ?
(ii)
If so, what is the description of such domain?
We answer (i) in the afﬁrmative, and answer (ii) by relating the domain of W to the set of M -matrices, which is deﬁned below.
Deﬁnition 1 ( M -matrix 1 ) .
An M -matrix is a matrix A ∈ R d
× d of the form A = sI − B , where B ≥and s > ρ ( B ) .
M -matrices were introduced by Ostrowski [39] and arise in a variety of areas including input-output analysis in economics, linear complementarity problems in operations research, ﬁnite difference methods for partial differential equations, and Markov chains in stochastic processes.
To the best of our knowledge, we are the ﬁrst to connect the notion of M -matrices to graphical model structure learning through an acyclicity characterization.
The following proposition is an immediate consequence of Deﬁnition 1:
Proposition 1 (Berman and Plemmons [5]) .
Let A ∈ R d × d be an M -matrix, then:
(i) (cid:60) ( λ i ( A )) > 0 , for all
i ∈
[ d ] .
(ii)
A − 1 exists and is nonnegative, i.e., A − 1 ≥ 0 .
In the above, item (i) states that the eigenvalues of an M -matrix lie in the open right-half plane.
Matrices which satisfy the latter property are also known as positive stable matrices.
We thus have that M -matrices are special cases of positive stable matrices.
It follows that the determinant of any
M -matrix is positive.
This fact will be used for deﬁning h s ldet ( W ) , our acyclicity characterization given in Theorem 1.
Finally, the nonnegativity of the inverse from item (ii) will be used to understand some properties of the gradient of h s ldet ( W ) .
We now deﬁne the domain over which (4) =
⇒ (3) (see Theorem 1).
For any s > 0 , deﬁne
[Equation]
i.e. W s is the set of real matrices whose entry-wise square given by W ◦ W have spectral radius less than s .
The following lemma lists some relevant properties of W s .
Lemma 1.
Let W s deﬁned as in (5) .
Then, for all s > 0 :
(i) DAGs ⊂
W s .
(ii) W s is path-connected.
(iii) W s ⊂ W t for any t > s .
In Lemma 1, item (i) implies that the W we look for is in the interior of W s ; item (ii) indicates that we can ﬁnd a path from any point in W s to any DAG without leaving the set W s ; item (iii) shows that one can vary s to enlarge or shrink the set W s .
Having deﬁned the domain set W s , we now deﬁne our acyclicity characterization h s ldet : W s → R .
Recall that item (i) in Proposition 1 implies that applying the logarithm function to the determinant of an M -matrix is always well deﬁned, which motivates our following result.
Theorem 1 (Log-determinant characterization) .
Let s > 0 and let h s ldet : W s → R be deﬁned as h s ldet ( W ) def = − log det( sI − W ◦ W ) + d log s .
Then, the following holds:
(i) h s ldet ( W ) ≥ 0 , with h s ldet ( W ) =
if and only if W is a DAG.
(ii) ∇ h s ldet ( W ) = 2( sI − W ◦ W ) −(cid:62) ◦ W , with ∇ h s ldet ( W )
=
if and only if W is a DAG.
3.1 Properties of h s ldet ( W )
In this section, we list several properties of our acyclicity characterization.
The ﬁrst property we discuss is related to the entries of ∇ h s ldet ( W ) .
Lemma 2.
For all i, j ∈ [ d ] , [ ∇ h s ldet ( W )] i,j = 0 if and only if W i,j = 0 or there is no directed walk from j to i .
Equivalently, [ ∇ h s ldet ( W )] i,j (cid:54)
=
if and only if the edge i → j is part of some cycle in
W .
Finally, whenever [ ∇ h s ldet ( W )] i,j (cid:54) = 0 , we have that sign([ ∇ h s ldet ( W )] i,j )
= sign( W i,j ) .
The lemma above characterizes the nonzero entries and their signs of the gradient of h s ldet .
This property formally offers a regularizer perspective for h s ldet , which we highlight next.
Remark 2 (A regularizer viewpoint) .
The function h s ldet promotes small parameters values in much the same way that the classical (cid:96) 1 and (cid:96) 2 regularizers do.
In contrast to the latter regularizers, h s ldet will only shrink the value of a parameter W i,j if and only if the edge ( i, j ) is part of some cycle in W , as prescribed by Lemma 2.
Recall that h expm ( W ) =
Tr( e W ◦ W ) − d
and h poly ( W ) =
Tr(( I + 1 d W ◦ W ) d ) − d .
It was noted by [54] that acyclicity characterizations of the form
Tr( (cid:80) dp
=1 c p ( W ◦ W ) d ) for c p > 0 also have the property in Lemma 2.
This implies that h expm and h poly hold the property above and can also be interpreted as acyclicity regularizers.
We note that the interesting part here is that h s ldet holds this property besides being different in nature to h expm and h poly .
Next, we state an important consequence of Lemma 2, which is related to the direction of ∇ h s ldet ( W ) .
Corollary 1.
At any W ∈ W s , the negative gradient ∇ h s ldet ( W ) points towards the interior of W s .
In optimization, the Hessian matrix plays an important role as it contains relevant information about saddle points and local extrema of a function, and is key to Newton-type methods.
Another appealing property of h s ldet ( W ) is that it has a Hessian described by a simple closed-form expression.
Lemma 3.
The Hessian of h s ldet ( W ) , which resides in R d 2 × d 2 , is given by:
[Equation]
where N = ( sI − W ◦ W ) − 1 , ⊗ denotes the Kronecker product, and K dd is the d 2 × d 2 commutation matrix such that K dd vec( A ) =
vec( A (cid:62) ) , for any d × d matrix A .
Here we note that among h expm , h poly and h s ldet , only h s ldet has a tractable expression for the Hessian.
Furthermore, note that ∇ 2 h s ldet ( W ) is indexed by vertex pairs so that the entry [ ∇ 2 h s ldet ( W )]
( k,l ) , ( p,q ) corresponds to the second partial derivative
[Equation]
Using Lemma 3, we can characterize the nonzero entries, and their signs, of the Hessian of h s ldet .
Corollary 2.
The entries of the Hessian ∇ 2 h s ldet ( W ) are described as follows:
[Equation]
where N = ( sI − W ◦ W ) − 1 .
Moreover, an off-diagonal entry [ ∇ 2 h s ldet ( W )] ( k,l ) , ( p,q ) is nonzero if and only if there exists a cycle in W of the form q → p (cid:32) l → k (cid:32) q , and has a sign equal to sign( W l,k W q,p ) .
Lastly, a diagonal entry [ ∇ 2 h s ldet ( W )] ( k,l ) , ( k,l ) is nonzero if and only if there exists a directed walk from k to l , and its sign is always positive.
Recall from Theorem 1 that all DAGs attain the minimum value and are critical points of h s ldet ; thus,
DAGs are local (and global) minimum of h s ldet and the Hessian matrix evaluated at a DAG must be positive (semi)deﬁnite.
Let us corroborate the latter, when W is a DAG, from Corollary 2 we have that the off-diagonal elements of ∇ 2 h s ldet ( W ) are zero, while the diagonal entries are nonnegative.
That is, the Hessian is positive semideﬁnite whenever W is a DAG.
This implies that all its stationary points are global minima: Such functions are called invex [20, 30].
Corollary 3.
Let s > 0 .
Then, h s ldet ( W ) is an invex function, i.e., all its stationary points are global minima, and these correspond to DAGs.
We note that even though h expm and h poly are also invex functions, they were not explicitly considered as invex functions before.
In fact, it was noted in [57] that DAGs were global minima of h expm but no characterization of its stationary points were given.
Wei et al.
[54] noted that DAGs were stationary points of h expm and h poly but the notion of invexity was not explicitly stated.
Remark 3 (A dynamical system perspective) .
The importance of invexity here is that in the eyes of h s ldet , all DAGs are the same.
That is, DAGs correspond to the set of attractors in h s ldet , and depending on the initial condition, the system will converge to a different attractor.
This offers the following viewpoint for the role of the score function Q ( f ; X ) , namely, “use the score Q to ﬁnd a basin of attraction such that the force ﬁeld of h s ldet will dictate the trajectory towards a DAG that is equal or close to the ground-truth”.
In Figure 1, we illustrate in a toy example the properties discussed in this subsection.
[Figure]
3.2 Why the log-determinant regularizer is preferable to existing acyclicity regularizers
In this section, we present three arguments as to why one should use h s ldet instead of existing functions such as h expm and h poly .
We invite the reader to look at Appendix B for additional details.
Argument (i).
h s ldet does not diminish cycles of any length.
Let us expand the functions h expm and h poly in their sum of matrix powers form, that is, h expm ( W ) =
(cid:80) ∞
k =0 1 / k !
Tr(( W ◦ W ) k ) − d and h poly ( W ) =
(cid:80) d k =
( dk ) /
d k Tr(( W ◦ W ) k ) − d .
Recall also that the entry [( W ◦ W )
k
i,i represents the sum of weighted walks from node i to node i of length k , where each edge has weight w 2 u,v .
Thus, one can notice that if W has cycles of length k , their contribution to h expm and h poly are diminished by 1 / k !
and ( dk ) /
d k , respectively.
Numerically, the latter can be problematic for the following reason:
Cycles of length k can go undetected even for small values of d and k .
In practice, a value of h expm , h poly ∈ [10 − 8 , 10 − 5 ] is typically regarded as zero [54, 57].
Consider a cycle graph of d nodes where each edge weight is +1 or − 1 .
The plot in Figure 2 shows how the values of h expm and h poly decay much faster than that of h s ldet ; in fact, at d = 13 we already observe h expm ( W ) ≈ 10 − 9 and h poly ( W ) ≈ 10 − 14 , i.e., cycles of length at least 13 would be numerically undetected by h expm and h poly .
In contrast, we observe that the value of the log-det function remains bounded away from zero and is able to detect larger cycles.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
In contrast, letting s = 1 . 001 , we observe that h s ldet can stay away from zero even for a cycle graph of 100 nodes.
Finally, we note a similar pattern for the entrywise
(cid:96) ∞ norm of the gradients.
At ﬁrst glance, it might seem difﬁcult to directly compare the value of h s ldet to h expm and h poly .
We next show that when s = 1 , h s =1 ldet is an upper bound to h expm and h poly .
Lemma 4.
For all W ∈ W s =1 , we have h poly ( W ) ≤ h expm ( W ) ≤ h s =1 ldet ( W ) .
The lemma above shows that in spite of h s =1 ldet , h expm , and h poly being exact acyclicity characterizations, h s =1 ldet will attain the largest value.
Argument (ii).
h s ldet has better behaved gradients.
Similar to argument (i), we show in Appendix
B that h expm and h poly are susceptible to vanishing gradients even when the graph contains cycles (see Figure 2).
The following lemma states that the magnitude of each entry of ∇ h s ldet at least as large as the magnitude of the corresponding entry of ∇ h expm and ∇ h poly , and hence, h ldet has larger gradients to guide optimization.
Lemma 5.
For any walk of length k , its contribution to the gradients ∇ h expm ( W ) and ∇ h poly ( W ) are diminished by 1 / k !
and ( d − 1 k ) / ( d − 1) k , respectively.
In contrast, ∇ h s =1 ldet ( W ) does not diminish any walk of any length.
This implies that |∇ h poly ( W ) | ≤ |∇ h expm ( W ) | ≤ |∇ h s =1 ldet ( W ) | .
Argument (iii).
Computing h s ldet and ∇ h s ldet is empirically faster.
Even though h s ldet , h expm , and h poly all three share the same computational complexity of O
( d 3 ) , in practice h s ldet can be computed in about an order of magnitude faster than h expm and h poly .
In Figure 3, we compare the runtimes of h s ldet , h expm and h poly for randomly generated matrices, where we observe that computing h s ldet can be 10x faster than h expm and h poly .
See Appendix B for further details.
[Figure]
Optimization
In the previous section we argued why the log-det function should be preferred in practice.
Let f θ denote a model with parameters θ for the functions f j in (1), e.g., neural networks as in [58].
In this section we turn to the problem of minimizing a given score function Q ( f θ ; X ) constrained to h s ldet ( W ( θ ))
= 0 .
That is, we aim to solve:
[Equation]
subject
[Equation]
where we include the (cid:96) 1 regularizer to promote sparse solutions.
Since the inception of the purely continuous framework for learning DAGs in [57], almost all followup work uses the augmented Lagrangian (AML) scheme to tackle problem (6), and L-BFGS-B [38] for solving the sequence of unconstrained problems.
Motivated by the properties of h s ldet given in 3.1, we propose a simpler scheme named DAGMA, based on solving a sequence of unconstrained problems in which h s ldet is simply seen as a regularizer.
DAGMA resembles the central path approach of barrier methods [8, 38], or the classical path-following approach for solving lasso problems [e.g.
Our method is given in Algorithm 1.
Require: Data matrix X , initial central path coefﬁcient µ (0) (e.g., 1), decay factor α ∈ (0 , 1) (e.g., 0 . 1 ), (cid:96) 1 parameter β 1 > 0 (e.g., 0.01), log-det parameter s > 0 (e.g., 1), number of iterations T .
1: Initialize θ
so
W ( θ (0) )
∈ W s .
do 3:
Starting at θ ( t ) , solve θ (
+
) =
arg min θ µ ( t ) ( Q ( f θ ; X ) + β 1 (cid:107) θ (cid:107) 1 ) + h s ldet ( W ( θ )) 4:
Set µ ( t +1) =
αµ ( t )
Ensure: W ( θ ( T ) )
The following lemma states that DAGMA will return a DAG at the limit of the central path.
This is a critical distinction against existing methods, many of which rely on some type of post-processing (e.g. thresholding) to ensure that the solution is a DAG.
Lemma 6.
Algorithm 1 is guaranteed to return a DAG whenever µ ( t ) → 0 .
4.1 Practical Considerations
As in barrier methods, where it is required to start at the interior of the feasibility region, in Algorithm 1, we require that the initial point W ( θ (0) ) be inside W s .
This is very easy to achieve since the zero matrix is in the interior of W s for any s > 0 ; therefore, in our experiments we simply set θ (0) = 0 .
Note that in Algorithm 1, we let µ ( t ) decrease by a constant factor at each iteration; however, it is possible to specify explicitly the value of each µ ( t ) , e.g., for T = 4 , we can let µ = { 1 , 0 . 1 , 0 .
Regarding the choice of s , in principle s could take any value greater than zero since DAGs are inside W s for any s > 0 .
Similar to µ , it is also possible to let s vary at each iteration, e.g., for T = 4 , we can set s = { 1 , 0 .
In practive, we observe that slightly decreasing s can help to obtain larger gradients as W gets closer to a DAG.
Note, however, that letting s be equal or close to 1 is generally easier to optimize than setting s closer to zero, the reason being that for smaller values of s the volume of W s is smaller and will require much smaller learning rates to stay inside W s , hence affecting convergence.
Finally, we do not specify how to solve line 3 in Algorithm 1, this is because we leave the door open for different solvers to be used.
For our experiments in the next section, we solve line 3 by using a ﬁrst-order method with the ADAM optimizer [24], which works remarkably well as shown in our experiments.
It remains as future work to exploit the Hessian structure of h ldet given in Lemma 3 for second-order methods.
Experiments
We compare our method against GES [11], PC [50], NOTEARS [57], and GOLEM [37] on both linear and nonlinear SEM.
The source codes for each of the aforementioned methods are speciﬁed in
Appendix C. Consistent with previous work in this area (e.g., NOTEARS and follow up work), we have not performed any hyperparameter optimization: This is to avoid presenting unintentionally biased results.
As a concrete example, for each of the SEM settings, we simply chose a reasonable value for the (cid:96) 1 penalty coefﬁcient and used that same value for all ER and SF graphs across many different numbers of nodes.
Our experimental setting is similar to [57, 58].
Moreover, since the accuracy of certain methods were signiﬁcantly lower than other methods, we only report the result against competitive methods in the main text; full results for all settings and methods can be found in
Appendix C.
In the following ﬁgures, ER4 and SF4 denote the Erd˝os-Rényi and scale-free graphs, where, for each number of nodes d , each simulated graph has in expectation 4 d number of edges.
It is worth noting that the experimental settings by Zheng et al.
[57, 58] consider graph models such as ER1, ER2, ER4,
SF1, SF2, and SF4.
Here we mainly focus in the hardest settings, that is, ER4 and SF4 .
For the linear SEM setting, we present results for graphs with small to moderate number of nodes in Figure 4, and for graphs with large number of nodes in Figure 5.
In both regimes, we observe that DAGMA obtains signiﬁcant speedups and improvements in terms of structural accuracy against NOTEARS and GOLEM, even though GOLEM is speciﬁc to and specialized for linear Gaussian SEMs.
For the nonlinear setting in Figure 6, in the generalized linear model (logistic), we similarly observe that DAGMA attains major speedups and improvements on SHD against NOTEARS [57].
For the general nonlinear model (neural networks), we observe that DAGMA is comparable in SHD to the
NONLINEAR NOTEARS [58] but obtains about signiﬁcant speedups.
We invite the reader to look at Appendix C for more details and additional experiments
[Figure] [Figure]
Final Remarks
A relevant assumption in this work is that of sufﬁciency, that is, there are no hidden variables that are a common cause of at least two observed variables.
While this assumption is widely used for
[Figure]
c t u r a l
H a mm
i ng
D i s t an c e
[Figure]
l l l l l l ll l l l l
[Figure]
structure learning, we nonetheless highlight that in practice it is very difﬁcult to ﬁnd scenarios where such assumption holds.
As with all work that assumes sufﬁciency, our work is an important necessary step to understanding settings with hidden variables.
Finally, we note that the work by [6] proposes a differentiable approach for ADMG for the semi-Markovian case using h expm .
It is left for future work to explore the performance of such method using h ldet
[Figure]
Another important limitation of this and previous work on the continuous framework for learning
DAGs is that of providing guarantees on the learned structure.
As in real-life applications one does not have access to the ground-truth DAG, there is much uncertainty as to whether an edge in the predicted
DAG actually corresponds to a causal relation.
Thus, there is still a need for formal guarantees under the continuous framework.
Acknowledgments and Disclosure of Funding
This material is based upon work supported by the National Science Foundation under Grant # 2127309 to the Computing Research Association for the CIFellows 2021 Project.
We are also grateful for the support of the University of Chicago Research Computing Center for assistance with the calculations carried out in this work.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (See Section 6) (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (See Section D) (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (See Sectionand 3) (b) Did you include complete proofs of all theoretical results?
[Yes] (See Section A) 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (See Section C) (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (See Section C)
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (See Section C)
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] (See Section C) 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (See Section C) (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
SUPPLEMENTARY MATERIAL
DAGMA: Learning DAGs via M -matrices and a Log-Determinant Acyclicity Characterization
A Detailed Proofs A.1 Proof of Theorem
Proof.
We ﬁrst note that for any s > 0 and matrix A ∈ R d × d we have det( sA )
=
s d A.
Then, log det( sI − W ◦ W ) − d log s = log( s d det( I − s − 1 W ◦ W ))
− d log s = log det( I − s − 1 W ◦ W ) .
Moreover, since W ∈ W s , we have that s > ρ ( W ◦ W ) or equivalently 1 > ρ ( s − 1 W ◦ W ) .
Thus, in the sequel of the proof we set s = 1 w.l.o.g.
Item (ii).
The gradient expression follows from standard matrix calculus [46].
From Lemma 2, it follows that W is an stationary point of h s =1 ldet ( W ) , i.e., ∇ h s =1 ldet ( W ) = 0 , if and only if W corresponds to a DAG.
Item (i).
From the item above, we characterized the stationary points of h s =1 ldet .
Moreover, from
Proposition 1, we know that for any W ∈ W s , the gradient ∇ h s =1 ldet is well-deﬁned since I − W ◦ W is an M -matrix and, thus, its inverse exists.
Finally, note that at the boundary of W s , we have that h s =1 ldet ( W ) → ∞ .
From these observations, we have that the global minima of h s =1 ldet must be in the interior of W s and will correspond to the set of stationary points.
Hence, DAGs are local and global minima of h s =1 ldet .
We conclude by noting that if W is a DAG then we have det( I − W ◦ W )
= 1 and the equality h s =1 ldet ( W ) = 0 holds immediately.
Since DAGs are global minima, this implies that for all W ∈ W s we have h s =1 ldet ( W ) ≥ 0 .
A.2 Proof of Lemma
Proof.
Item (i).
The proof follows directly from the fact that the weighted adjacency matrix W of any DAG is a nilpotent matrix.
Since W ◦ W is also a nilpotent matrix, its spectral radius is zero, i.e., ρ ( W ◦ W ) = 0 .
Thus, for any s > 0 , if W is a DAG then W ∈ W s .
Item (ii).
Recall that a space X is path-connected if, for any two points x, y ∈ X , there exists a continuous function (path) φ : [0 , 1] → X such that φ (0)
=
x and φ (1) =
y .
Note that since the d × d zero matrix, 0 , has spectral radius zero, it clearly follows that 0 ∈ W s for any s > 0 .
We prove path-connectedness of W s by showing that for any W ∈ W s there exist a path φ to 0 .
Then, for any W ∈ W s , deﬁne φ ( t ) =
(1 − t ) W .
It is clear that φ is a continuous function on t , where φ (0)
=
W ∈ W s and φ (1) =
0 ∈ W s .
Now we need to show that φ ( t ) ∈ W s for all t ∈ (0 , 1) .
Let t 1 be an arbitrary number in (0 , 1) , by the nonnegativity of W ◦ W and by (1 − t 1 ) 2 < 1 , we have that (1 − t 1 ) 2 W ◦ W < W ◦ W .
Finally, by Perron-Frobenius theory on nonnegative matrices, we have that ρ (cid:0) (1 − t 1 ) 2 W ◦ W (cid:1) <
ρ ( W ◦ W ) <
s , where the last inequality follows by W ∈ W s , which implies that φ ( t 1 ) ∈ W s .
As the choice of t 1 was arbitrary, we conclude the proof.
Item (iii).
The proof follows immediately by the deﬁnition of W s .
A.3 Proof of Lemma
Proof.
First, recall that ∇ h s ldet ( W ) = 2( sI − W ◦ W ) −(cid:62) ◦ W .
Second, since ( sI − W ◦ W ) is an M -matrix, by Proposition 1, we have that ( sI − W ◦ W ) −(cid:62) ≥ 0 .
By the latter, whenever [ ∇ h s ldet ( W )] i,j (cid:54) = 0 , we have that sign([ ∇ h s ldet ( W )] i,j )
= sign( W i,j ) .
Finally, from the series expansion of the inverse we have:
[Equation]
by taking the transpose, that implies that the i, j entry [( sI − W ◦ W ) −(cid:62) ] i,j is nonzero if and only if there exists a directed walk from j to i .
By taking the Hadamard product, we have that [( sI − W ◦ W ) −(cid:62)
◦ W
i,j is nonzero if and only if W i,j (cid:54) =
and [( sI − W ◦ W ) −(cid:62) ]
i,j (cid:54) = 0 , i.e., there must exist a closed walk of the form i → j (cid:32) i .
Which concludes the proof.
A.4 Proof of Lemma
Proof.
We use the Magnus-Neudecker convention [28] for calculating the Hessian.
Then, by taking differentials and vectorizing, we obtain:
[Equation]
Recall that N = ( sI − W ◦ W ) − 1 , we now derive the expression for vec( ∂N (cid:62) ) ,
[Equation]
plugging in the last equality into eq.(7), we have
[Equation]
which concludes the proof.
A.5 Proof of Lemma
Proof.
The comparison between h expm and h poly is straightforward by looking at the coefﬁcients of their series expansions.
Recall that, h expm ( W ) = (cid:80) ∞ k =0 1 / k !
Tr(( W ◦ W ) k ) − d and h poly ( W ) =
[Equation]
dk =
[Equation] [Equation]
− d .
Since 1 / k ! ≥ ( dk ) /
d k , it is clear that h expm ( W ) ≥ h poly ( W ) .
To prove that h s =1 ldet ( W ) ≥ h expm ( W ) , we use the fact that every square matrix has a Jordan canonical form.
Let W ◦
W =
Q − 1 JQ , where Q is an invertible matrix and J is in Jordan normal form (i.e., a block diagonal matrix with 1 s in the super-diagonal), we have that Tr( e W ◦ W ) =
Tr( e J ) .
Let Λ( W ◦ W ) be the set of distinct generalized eigenvalues of W ◦ W , and m λ be the multiplicity corresponding to λ ∈ Λ( W ◦ W ) .
Then, we have that
[Equation]
From h s =1 ldet ( W ) we have,
[Equation]
where log denotes the principal branch of the complex logarithm.
For any complex λ , we have the
Taylor series:
[Equation]
where both series converges precisely for all complex numbers | λ | ≤ 1 , λ (cid:54)
= 1 , which is the case as W ∈ W s =1 .
From the latter, it is clear to see that h expm ( W ) ≤ h s =1 ldet ( W ) , which conludes the proof.
A.6 Proof of Lemma
Proof.
First let us write the gradients for the different acyclicity characterizations.
Then, we have ∇ h expm ( W ) =
2( e W ◦ W ) (cid:62) ◦ W , ∇ h poly ( W ) =
2(( I + 1 d W ◦ W )
d − 1 ) (cid:62) ◦ W , and ∇ h s =1 ldet ( W ) =
(( I − W ◦ W ) − 1 ) (cid:62) ◦ W .
When taking absolute values, it is clear that they differ due to the left-hand side of each Hadamard product.
Thus, we need to look at the entries of: | e W ◦ W | , | ( I + 1 d W ◦ W )
d − 1 | , and | ( I − W ◦ W ) − 1 | .
From their series expansions we have:
[Equation]
Since W ◦ W is nonnegative, each power ( W ◦ W ) k is also nonnegative.
Therefore, by noting that ( d − 1 k ) /
( d − 1) k ≤ 1 / k ! ≤ 1 for all k , the statement follows.
A.7 Proof of Lemma
Proof.
The proof follows by noting that at the limit of the central path ( µ ( t ) → 0 ) we solve the following problem:
[Equation]
Then, by the invexity property of h s ldet (see Corollary 3), it follows that the solution W ( (cid:98) θ ) must be
a
DAG.
A.8 Proof of Corollary
Proof.
For any W ∈ W s , from Lemma 2, we know that the nonzeros of ∇ h s ldet ( W ) have the same sign as the corresponding entries in W .
Then, let Y
=
W − a ∇ h s ldet ( W ) for a small value a such that | Y | ≤ | W | .
It follows that Y ◦ Y ≤ W ◦ W .
Since Y ◦ Y and W ◦ W are nonnegative matrices, by Perron-Frobenius, we have that ρ ( Y ◦ Y ) ≤
ρ ( W ◦ W ) .
The latter implies that Y ∈ W s , thus, the negative gradient, −∇ h s ldet , must point towards the interior of W s .
A.9 Proof of Corollary
Proof.
Given the Hessian expression in Lemma 3, the expressions for its entries follow by simple algebraic manipulation.
From the argument in the proof of Lemma 2 (see Apeendix A.3), we have that N i,j >
if and only if there exist a directed walk from i to j .
By the latter, the signs and cycle interpretations follow.
A.10 Proof of Corollary
Proof.
Follows from Theorem 1.
B Additional Discussions B.1 Additional Example for Section 3.
In Figure 1, we provided an example of a two-node graph to visualize the properties of h ldet .
In Figure 7, we present another example for a three-node graph with three edges (parameters).
Speciﬁcally, the graph is parameterized by W =
[Equation]
 .
Here note that for W to be a DAG at least one of w 2 or w 3 must be zero.
[Figure]
B.2 Further Details for Section 3.
In this section, we expand the discussion given in Section 3.
Argument (i).
h s ldet does not diminish cycles of any length.
Let h tinv ( W ) =
Tr(( I − W ◦
[Equation]
−
[Equation]
=
[Equation]
∞ k =
Tr(( W ◦ W ) k ) − d .
We note that h tinv is another acyclicity characterization, previously considered by Zheng et al.
From its series expansion, the reader might wonder if simply considering h tinv ( W ) would be a better alternative than h ldet ( W ) , h expm ( W ) , and h poly ( W ) .
The answer is no : While this alternative characterization does not suffer from the dimishing cycle problem, it is prone to numerical instability, which was already noted in [57].
This numerical instability is mainly due to the exploding gradients of Tr(( I − W ◦ W ) − 1 ) , which we discuss in our next argument.
Argument (ii).
h s ldet has better behaved gradients.
Let us write the gradients of the different acyclicity characterizations.
We have:
[Equation]
From the series expansion for ( I − W ◦ W ) − 2 we obtain:
[Equation]
Then, ∇ h tinv is numerically unstable whenever W ◦ W has walks of moderate weight since all walks of length k are now weighted by k + 1 , thus, being prone to exploding gradients.
Similar to argument (i), we have the following series expansions:
[Equation]
and
[Equation]
One can observe that for s = 1 , the gradient of the log-determinant weights equally all cycles of any length, whereas ∇ h expm and ∇ h poly are again susceptible to the vanishing cycle problem and, thus, they might suffer from vanishing gradients.
The plot in Figure 2 shows how the gradients of h expm and h poly decay at a very fast rate as the cycle graph has more nodes.
Argument (iii).
Computing h s ldet and ∇ h s ldet is empirically faster.
In Figure 3, we compared the runtimes of h s ldet , h expm and h poly for randomly generated matrices.
We used the benchmarking library from PyTorch for better runtime estimates over single threads.
Experiments were conducted on an Intel Xeon processor E5 v4 with 2.40 GHz frequency.
The reason that computing h s ldet and ∇ h s ldet is faster is that it involves computing a log-determinant and a matrix inverse respectively, and both of these problems enjoy the large body of work on optimized libraries for matrix factorizations (e.g., LU decomposition) and linear-system solvers.
In contrast, computing h expm relies on a truncated Taylor series of the exponential whose degree is typically estimated using scaling [22, 1], and this requires several matrix-matrix multiplications.
The matrix exponential is also a notoriously tricky object to compute [33].
Similar to h expm , computing h poly also requires several matrix-matrix multiplications and, thus, both attain similar performances.
C Detailed Experiments
Computing.
All experiments were conducted on an 8-core Intel Xeon processor E5-2680v4 with 2.40 GHz frequency, and 32GB of memory.
Each experiment had a wall time of 36 hours.
Graph Models.
Each simulation in our experiments samples a graph from two well-known random graph models:
• Erdos-Rényi (ER) graphs: These are random graphs whose edges are added independently with equal probability.
We use the notation ER k to indicate that the graph model is an ER graph with kd edges in expectation.
• Scale-free (SF) graphs: These are random graphs simulated according to the preferential attachment process [4].
We use the notation SF k to indicate that the graph model is an SF graph with kd edges in expectation and β = 1 , where β is the exponent used in the preferential attachment process.
It is worth noting that since we consider directed graphs, this particular model corresponds to Price’s model, a classical graph model for the growth of citation
networks
Note that ER graphs are random undirected graphs.
To produce a DAG, we draw a random permutation of numbers from 1 to d and orient the edges respecting this vertex ordering.
For the case of SF graphs, the edges are oriented each time a new node is attached, thus, the sampled graph is a DAG.
Once the ground-truth DAG is generated, we need to simulate the structural equation model, where we provide experiments for linear and nonlinear SEMs.
Remark 4.
It is worth noting that the experimental settings by Zheng et al.
[57, 58] consider graph models such as ER1, ER2, ER4, SF1, SF2, and SF4.
Here we mainly focus in the hardest settings, that is, ER4 and SF4 .
Metrics.
We evaluate the performance of each algorithm with the following four metrics:
• Structure Hamming distance (SHD) :
A standard measurement for structure learning that counts the total number of edges additions, deletions, and reversals needed to convert the estimated graph into the true graph.
• True Positive Rate (TPR) :
Measures the proportion of correctly identiﬁed edges with respect to the total number of edges in the ground-truth DAG.
• False Positive Rate (FPR) :
Measures the proportion of incorrectly identiﬁed edges with respect to the total number of absent edges in the ground-truth DAG.
• Runtime : Measures how much time the algorithm takes to run, we use it to measure the speed of the algorithms.
Remark 5.
Consistent with previous work in this area (e.g. NOTEARS and its follow-ups), we have not performed any hyperparameter optimization: This is to avoid presenting unintentionally biased results.
As a concrete example, for each of the following SEM settings, we simply chose a reasonable value for the (cid:96) 1 penalty coefﬁcient and used that same value for all ER and SF graphs across many different numbers of nodes.
C.1 SEM: Linear Setting
In the linear case, the functions f j in (1) are directly parameterized by the weighted adjacency matrix
W .
That is, we have the following set of equations:
[Equation]
where W =
[ w 1 | · · · | w d ] ∈ R d × d , and Z j ∈ R represents the noise.
Here W encodes the graphical structure, i.e., there is an edge X k
→ X j if and only if W k,j (cid:54) = 0 .
Then, given the ground-truth DAG B ∈ { 0 , 1 } d × d from one of the two graph models ER or SF, we assigned edge weights independently from Unif (cid:0) [ − 2 , − 0 .
5] ∪ [0 . 5 , 2] (cid:1) to obtain a weight matrix
W ∈ R d × d .
Given W , we sampled X = W (cid:62) X + Z ∈ R d according to the following three noise models:
• Gaussian noise: Z j ∼ N (0 , 1) , ∀ j ∈
[ d ] .
• Exponential noise: Z j ∼ Exp(1) , ∀ j ∈ [ d ] .
• Gumbel noise: Z j ∼ Gumbel(0 , 1) , ∀ j ∈
[ d ] .
Based on these models, we generated random datasets X ∈ R n × d by generating the rows i.i.d.
according to one of the models above.
For each simulation, we generated n = 1000 samples, unless otherwise stated.
To measure the quality of a model, we use the least-square loss
[Equation]
8) where (cid:107)·(cid:107) F denotes the Frobenius norm.
The coefﬁcient β 1 used for (cid:96) 1 regularization, and other parameters required for DAGMA (see Algorithm 1), are later speciﬁed for each ﬁgure.
The implementation details of the baselines are listed below:
• GES (speciﬁcally, the FGES algorithm in [47]) and PC [50] are standard baselines for structure learning.
Their implementation is based on the py-causal package, available at https://github.com/bd2kccd/py-causal .
The exact set of hyperparameters used are: – For PC: testId = ‘fisher-z-test’, depth = 3, fasRule = 2, dataType = ‘continuous’, conflictRule = 1, concurrentFAS =
True, useMaxPOrientationHeuristic = True .
– For GES: scoreId = ‘cg-bic-score’, maxDegree = 5, dataType = ‘continuous’, faithfulnessAssumed = False .
• The NOTEARS method in Zheng et al.
[57] was implemented using the author’s
Python code available at: https://github.com/xunzheng/notears .
Its score function is also the least square as deﬁned in eq.(8).
For the (cid:96) 1 coefﬁcient, for a fair comparison, we use the same value used for DAGMA.
For the rest of hyperparameters, we use their default values.
• The GOLEM method in Ng et al.
[37] was implemented using the author’s
Python and Tensorﬂow code available at: https://github.com/ignavierng/golem .
Here we use their default set of hyperparameters, that is, λ 1 = 0 .
02 and λ 2 = 5 , for other details of their method we refer the reader to Appendix F of Ng et al.
C.1.1 Small to Moderate Number of Nodes
Following the aforementioned process to generate data, in this section, we test the methods for graphs with number of variables d ∈ { 20 , 30 , 50 , 80 , 100 } .
We use the following setting for DAGMA (Algorithm 1): Number of iterations T = 4 , initial central path coefﬁcient µ (0) = 1 , decay factor α = 0 .
1 , (cid:96) 1 coefﬁcient β 1 = 0 .
05 , log-det parameter s = { 1 , .
For each problem in lineof Algorithm 1, we implement an adaptive gradient method using the ADAM optimizer [24].
The hyperparameters for ADAM are: Learning rate of 3 × 10 − 4 , and ( β 1 , β 2 ) =
For t = { 0 , 1 , 2 } , we run ADAM for 2 × 10 4 iterations or until the loss converges, whichever comes ﬁrst.
For t = 3 , we run ADAM for 7 × 10 4 iterations or until the loss converges, whichever comes ﬁrst.
We consider that the loss converges if the relative error between subsequent iterations is less than 10 − 6 .
Finally, as in previous work including the baseline methods [57, 58, 37], a ﬁnal thresholding step is performed as it was shown to help reduce the number of false discoveries.
For all cases, we use a threshold of 0 . 3 .
The results for different graph models (ER4, SF4), and different noise distributions (Gaussian,
Gumbel, exponential), are shown in Figure 8.
In Table 1, we average the SHDs and runtimes across graph and noise types, for the competitive methods.
Here we note in particular that for d = 100 ,
DAGMA obtains an improvement of 74 . 9% and 76 .
5% in SHD against GOLEM and NOTEARS, respectively; also, DAGMA runs 7 . 7 and 19 .
1 times faster than GOLEM and NOTEARS, respectively.
Finally, we note that DAGMA performs much better than GOLEM besides the latter being tailored to linear Gaussian models.
most competitive methods.
Each metric was averaged across different graph and noise types.
Finally, the errors denote 95% conﬁdence intervals on 10 repetitions.
[Table]
C.1.2 Large Number of Nodes
In this section
[Table]
we test DAGMA
[Table]
GOLEM
[Table]
and NOTEARS for graphs with number of variables d ∈ { 200 , 300 , 500 , 800 , 1000 } .
We do not test PC and GES as they are not competitive in terms of accuracy, as shown in Figure 8.
We follow the same setting for DAGMA given in Section C.1.1.
The results for different graph models (ER4, SF4), and different noise distributions (Gaussian,
Gumbel, exponential), are shown in Figure 9.
In Table 2, we average the SHDs and runtimes across graph and noise types.
Here we note in particular that for d = 800 , DAGMA obtains an improvement of 90 . 2% and 65 . 5% in SHD against GOLEM and NOTEARS, respectively; also, DAGMA runs 6 . 2 and 25 times faster than GOLEM and NOTEARS, respectively.
For d = 1000 , we observe that NOTEARS takes more than 36 hours, which is the reason we could not report its performance.
Finally, we note that once again DAGMA performs much better than GOLEM besides the latter being tailored to linear Gaussian models.
[Figure]
t r u c t u r a l
H a mm
i ng
[Figure]
i s t an
[Figure]
e
[Figure]
ue
[Figure]
i t e
R a t e ( % )
[Figure]
C.1.3 DAGMA vs GOLEM in Sparser Graphs
We note that in the work by Ng et al.
[37], the authors performed experiments on large number of nodes only for ER2 graphs , that is, sparser graphs.
It was reported in Ng et al.
[37] that
their
[Table]
GOLEM method performed reasonably well for large number of nodes.
However, as shown in
Figure 9, for denser graphs such as ER4 and SF4, the performance of GOLEM degrades very fast as d increases.
In this section
[Table]
we experiment with the same graph model as in [37], i
[Table]
e., ER2, for d ∈ { 200 , 300 , 500 , 800 , 1000 , 2000 } We note that even though GOLEM is competitive in this regime, DAGMA still obtains signiﬁcant improvements.
DAGMA was run under the same setting described in Section C.1.1.
The results are reported in
Figure 10.
Here we note in particular that for d = 1000 , DAGMA obtains an improvement of 22 . 5% in SHD , and runs 8 .
5 times faster than GOLEM, even though the latter is customized for linear models.
For d = 2000 , GOLEM took more than 36 hours per simulation, hence, we could not report its performance.
C.1.4 DAGMA vs NOTEARS and GOLEM in Denser Graphs
For completeness, we run experiments on a denser graph type such as ER6 with Gaussian noise, for d ∈ { 20 , 40 , 60 , 80 , 100 } .
DAGMA was run under the same setting described in Section C.1.1.
The results are reported in Figure 11.
Here we note in particular that for d = 100 , DAGMA obtains an improvement of 73 .
1% and 44 .
5% in SHD against GOLEM and NOTEARS, respectively; also,
DAGMA runs 4 . 8 and 20 .
6 times faster than GOLEM and NOTEARS, respectively.
Finally, we note that even in the regime of small number of variables, GOLEM’s performance degrades very fast for denser graphs, while DAGMA’s performance remains the best among the three methods.
C.2 SEM: Non-Linear Setting C.2.1 Logistic Model
Given the ground-truth DAG B ∈ { 0 , 1 } d × d from an ER4 graph, we assigned edge weights independently from Unif (cid:0) [ − 2 , − 0 . 5] ∪ [0 . 5 , 2] (cid:1) to obtain a weight matrix W ∈ R d × d .
Given W , we sampled X j = Bernoulli (exp( w (cid:62) j X ) / (1 + exp( w (cid:62) j X ))) , ∀ j ∈ [ d ] .
Based on this model, we generated random datasets X ∈ R n × d by generating the i.i.d.
rows.
For each simulation, we generated n = 5000 samples for graphs with d ∈ { 20 , 40 , 60 , 80 , 100 , 200 , 400 , 600 , 800 , 1000 } nodes.
To measure the quality of a model, we use the log-likelihood loss
[Equation]
The implementation details of the baselines are listed below:
[Figure]
u r a l
[Figure]
a
[Figure]
i
[Figure]
i
[Figure]
t an c
[Figure]
• GES (speciﬁcally, the FGES algorithm in [47]) and PC [50] are standard baselines for structure learning.
Their implementation is based on the py-causal package, available at https://github.com/bd2kccd/py-causal .
The exact set of hyperparameters used are:
[Figure]
[Figure] [Figure]
– For PC: testId = ‘disc-bic-test’, depth = 4, fasRule = 2, dataType = ‘discrete’, conflictRule = 1, concurrentFAS =
True, useMaxPOrientationHeuristic = True .
– For GES: scoreId = ‘bdeu-score’, maxDegree = 5, dataType = ‘discrete’, faithfulnessAssumed = False .
• The NOTEARS method in Zheng et al.
[57] was implemented using the author’s
Python code available at: https://github.com/xunzheng/notears .
Its score function is also the log-likelihood loss as
deﬁned in eq.(9).
For the (cid:96) 1 coefﬁcient, for a fair comparison, we use the same value used for DAGMA.
For the rest of hyperparameters, we use their default values.
We use the following setting for DAGMA (Algorithm 1): Number of iterations T = 4 , initial central path coefﬁcient µ (0) = 10 , decay factor α = 0 .
1 , (cid:96) 1 coefﬁcient β 1 = 0 .
01 , log-det parameter s = { 1 , .
For each problem in line 3 of Algorithm 1, we implement an adaptive gradient method using the ADAM optimizer [24].
The hyperparameters for ADAM are: Learning rate of 3 × 10 − 4 , and ( β 1 , β 2 ) =
For t = { 0 , 1 , 2 } , we run ADAM for 10 4 iterations or until the loss converges, whichever comes ﬁrst.
For t = 3 , we run ADAM for 5 × 10 4 iterations or until the loss converges, whichever comes ﬁrst.
We consider that the loss converges if the relative error between subsequent iterations is less than 10 − 6 .
Finally, as in [57, 58, 37], a ﬁnal thresholding step is performed as it was shown to help reduce the number of false discoveries.
For all cases, we use a threshold of 0 . 3 .
The results are shown in Figure 12.
We note that for d = 1000 , DAGMA obtains an improvement of 60% in SHD and runs 4 .
8 times faster than NOTEARS.
Finally, we note that GOLEM is not considered for the nonlinear models as it only works for linear ones.
Neural Networks
[Figure]
t r u c t u r a l H a mm
i ng
D i s t an c e
[Figure]
are
C.2.2 Neural Network Model
We mainly follow the nonlinear setting of Zheng et al.
That is, given a ground-truth graph G , we simulate the SEM:
[Equation]
where Z j ∼ N (0 , 1) is a standard Gaussian noise.
Here f j is a randomly initialized multilayer perceptron (MLP) with one hidden layer of size 100 and sigmoid activation.
Similar to previous experiments, we generate a dataset X ∈ R n × d , with n = 1000 i.i.d.
samples.
For this setting, we only compare to NONLINEAR NOTEARS [58].
We refer the reader to [58] for a comprehensive comparison with other baselines.
For NONLINEAR NOTEARS and DAGMA, each f θ is modeled by a MLP with one hidden layer of size 10 and sigmoid activation.
In contrast to the original implementation of NONLINEAR NOTEARS [58] which uses the square loss, we use the log-likelihood as in [9] as we observe better performances for both methods.
We use the following setting for DAGMA (Algorithm 1): Number of iterations T = 4 , initial central path coefﬁcient µ (0) = 0 . 1 , decay factor α = 0 .
1 , (cid:96) 1 coefﬁcient β 1 = 0 .
02 , log-det parameter s = 1 .
For each problem in line 3 of Algorithm 1, we implement an adaptive gradient method using the ADAM optimizer [24].
The hyperparameters for ADAM are: Learning rate of 2 × 10 − 4 , and ( β 1 , β 2 ) =
For t = { 0 , 1 , 2 } , we run ADAM for 7 × 10 4 iterations or until the loss converges, whichever comes ﬁrst.
For t = 3 , we run ADAM for 8 × 10 4 iterations or until the loss converges, whichever comes ﬁrst.
We consider that the loss converges if the relative error between subsequent iterations is less than 10 − 6 .
Finally, as in [57, 58, 37], a ﬁnal thresholding step is performed as it was shown to help reduce the number of false discoveries.
For all cases, we use a threshold of 0 . 3 .
The results are shown in Figure 13.
We note that DAGMA and NOTEARS obtain similar performances in SHD; however, DAGMA can obtain 3x to 10x speedups over NONLINEAR NOTEARS.
Finally, we note that GOLEM is not considered for the nonlinear models as it only works for linear ones.
D Broader Impacts
A potential misuse of this type of work would be to purposely (or not) run the method proposed on a dataset that is biased.
Since we do not formally deal with inherent biases in the dataset (e.g., unfairness due to selection bias), it is possible to learn relationships that are not present in reality.
A user can then (un)intentionally report a result incorrectly claiming to have found the cause of a certain variable, thus, creating misinformation.