Original: G RADIENT -B ASED N EURAL DAG L EARNING
Revised: G RADIENT -B ASED N EURAL DAG L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel score-based approach to learning a directed acyclic graph (DAG) from observational data.
Revised: We propose a novel score-based approach to learning a directed acyclic graph (DAG) from observational data.

Original: We adapt a recently proposed continuous constrained optimization formulation to allow for nonlinear relationships between variables using neural networks.
Revised: We adapt a recently proposed continuous constrained optimization formulation to allow for nonlinear relationships between variables using neural networks.

Original: This extension allows to model complex interactions while avoiding the combinatorial nature of the problem.
Revised: This extension allows to model complex interactions while avoiding the combinatorial nature of the problem.

Original: In addition to comparing our method to existing continuous optimization methods, we provide missing empirical comparisons to nonlinear greedy search methods.
Revised: In addition to comparing our method to existing continuous optimization methods, we provide missing empirical comparisons to nonlinear greedy search methods.

Original: On both synthetic and real-world data sets, this new method outperforms current continuous methods on most tasks, while being competitive with existing greedy search methods on important metrics for causal inference.
Revised: On both synthetic and real-world data sets, this new method outperforms current continuous methods on most tasks, while being competitive with existing greedy search methods on important metrics for causal inference.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Structure learning and causal inference have many important applications in different areas of science such as genetics (Koller & Friedman, 2009; Peters et al., 2017), biology (Sachs et al., 2005) and economics (Pearl, 2009).
Revised: Structure learning and causal inference have many important applications in different areas of science such as genetics (Koller & Friedman, 2009; Peters et al., 2017), biology (Sachs et al., 2005) and economics (Pearl, 2009).

Original: Bayesian networks (BN), which encode conditional independencies using directed acyclic graphs (DAG), are powerful models which are both interpretable and computationally tractable.
Revised: Bayesian networks (BN), which encode conditional independencies using directed acyclic graphs (DAG), are powerful models which are both interpretable and computationally tractable.

Original: Causal graphical models (CGM) (Peters et al., 2017) are BNs which support interventional queries like: What will happen if someone external to the system intervenes on variable X?
Revised: Causal graphical models (CGM) (Peters et al., 2017) are BNs which support interventional queries like: What will happen if someone external to the system intervenes on variable X?

Original: Recent work suggests that causality could partially solve challenges faced by current machine learning systems such as robustness to out-of-distribution samples, adaptability and explainability (Pearl, 2019; Magliacane et al., 2018).
Revised: Recent work suggests that causality could partially solve challenges faced by current machine learning systems such as robustness to out-of-distribution samples, adaptability and explainability (Pearl, 2019; Magliacane et al., 2018).

Original: However, structure and causal learning are daunting tasks due to both the combinatorial nature of the space of structures (the number of DAGs grows super exponentially with the number of nodes) and the question of structure identiﬁability (see Section 2.2).
Revised: However, structure and causal learning are daunting tasks due to both the combinatorial nature of the space of structures (the number of DAGs grows super exponentially with the number of nodes) and the question of structure identiﬁability (see Section 2.2).

Original: Nevertheless, these graphical models known qualities and promises of improvement for machine intelligence renders the quest for structure/causal learning appealing.
Revised: Nevertheless, these graphical models known qualities and promises of improvement for machine intelligence renders the quest for structure/causal learning appealing.

Original: The usual motivation for learning a causal graphical model is to predict the effect of various interventions.
Revised: The typical motivation for learning a causal graphical model is to predict the effect of various interventions.

Original: A CGM can be best estimated when given interventional data, but interventions are often costly or impossible to obtained.
Revised: A CGM can be best estimated when given interventional data, but interventions are often costly or impossible to obtained.

Original: One approach is to use exclusively observational data andto rely on different assumptions which make the graph identiﬁable from the distribution (see Section 2.2).
Revised: As an alternative, one can use exclusively observational data and rely on different assumptions which make the graph identiﬁable from the distribution (see Section 2.2).

Original: This is the approach employed in this paper.
Revised: This is the approach employed in this paper.

Original: We propose a score-based method (Koller & Friedman, 2009) for structure learning named GraNDAG which makes use of a recent reformulation of the original combinatorial problem of ﬁnding an optimal DAG into a continuous constrained optimization problem .
Revised: We propose a score-based method (Koller & Friedman, 2009) for structure learning named GraNDAG which makes use of a recent reformulation of the original combinatorial problem of ﬁnding an optimal DAG into a continuous constrained optimization problem .

Original: In the original method named
Revised: In the original method named

Original: NOTEARS (Zheng et al., 2018), the directed graph is encoded as a weighted adjacency matrix which represents coefﬁcients in a linear structural equation model (SEM) (Pearl, 2009) (see Section 2.3) and enforces acyclicity using a constraint which is both efﬁciently computable and easily differentiable, thus allowing the use of numerical solvers.
Revised: NOTEARS (Zheng et al., 2018), the directed graph is encoded as a weighted adjacency matrix which represents coefﬁcients in a linear structural equation model (SEM) (Pearl, 2009) (see Section 2.3) and enforces acyclicity using a constraint which is both efﬁciently computable and easily differentiable, thus allowing the use of numerical solvers.

Original: This continuous approach improved upon popular methods while avoiding the design of greedy algorithms based on heuristics.
Revised: This continuous approach improved upon popular methods while avoiding the design of greedy algorithms based on heuristics.

Original: Our ﬁrst contribution is to extend the framework of Zheng et al.
Revised: Our ﬁrst contribution is to extend the framework of Zheng et al.

Original: (2018) to deal with nonlinear relationships between variables using neural networks (NN) (Goodfellow et al., 2016).
Revised: (2018) to deal with nonlinear relationships between variables using neural networks (NN) (Goodfellow et al., 2016).

Original: To adapt the acyclicity constraint to our nonlinear model, we use an argument similar to what is used in Zheng et al.
Revised: To adapt the acyclicity constraint to our nonlinear model, we use an argument similar to what is used in Zheng et al.

Original: and apply it ﬁrst at the level of neural network paths and then at the level of graph paths .
Revised: and apply it ﬁrst at the level of neural network paths and then at the level of graph paths .

Original: Although GraN-DAG is general enough to deal with a large variety of parametric families of conditional probability distributions (see Section 3.3 for caveats), our experiments focus on the 
Revised: Although GraN-DAG is general enough to deal with a large variety of parametric families of conditional probability distributions, our experiments focus on the special case of nonlinear

Original: We provide an implementation of GraN-DAG here.

Original: special case of nonlinear Gaussian additive noise models , since under speciﬁc assumptions, it provides appealing theoretical guarantees easing the comparison to other graph search procedures (see Section 2.2 & 3.3).
Revised: Gaussian additive noise models since, under speciﬁc assumptions, it provides appealing theoretical guarantees easing the comparison to other graph search procedures (see Section 2.2 & 3.3).

Original: On both synthetic and real-world tasks, we show GraN-DAG often outperforms other approaches which leverage the continuous paradigm, including DAG-GNN (Yu et al., 2019), a recent nonlinear extension of Zheng et al.
Revised: On both synthetic and real-world tasks, we show GraN-DAG often outperforms other approaches which leverage the continuous paradigm, including DAG-GNN (Yu et al., 2019), a recent nonlinear extension of Zheng et al.

Original: (2018) which uses an evidence lower bound as score.
Revised: (2018) which uses an evidence lower bound as score.

Original: Our second contribution is to provide a missing empirical comparison to existing methods that support nonlinear relationships but tackle the optimization problem in its discrete form using greedy search procedures, namely CAM (B¨uhlmann et al., 2014) and GSF (Huang et al., 2018).
Revised: Our second contribution is to provide a missing empirical comparison to existing methods that support nonlinear relationships but tackle the optimization problem in its discrete form using greedy search procedures, namely CAM (B¨uhlmann et al., 2014) and GSF (Huang et al., 2018).

Original: We show that GraN-DAG is competitive on the wide range of tasks we considered, while using similar pre and post-processing steps as CAM. 
Revised: We show that GraN-DAG is competitive on the wide range of tasks we considered, while using pre- and post-processing steps similar to CAM. We provide an implementation of GraN-DAG here.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Before presenting GraN-DAG, we review concepts relevant to structure and causal learning.
Revised: Before presenting GraN-DAG, we review concepts relevant to structure and causal learning.

Original: 2. C AUSAL GRAPHICAL MODELS
Revised: 2. C AUSAL GRAPHICAL MODELS

Original: We suppose the natural phenomenon of interest can be described by a random vector X ∈ R d entailed by an underlying CGM ( P X , G ) where P X is a probability distribution over X and G =
Revised: We suppose the natural phenomenon of interest can be described by a random vector X ∈ R d entailed by an underlying CGM ( P X , G ) where P X is a probability distribution over X and G =

Original: ( V, E ) is a DAG (Peters et al., 2017).
Revised: ( V, E ) is a DAG (Peters et al., 2017).

Original: Each node j ∈ V corresponds to exactly one variable in the system.
Revised: Each node j ∈ V corresponds to exactly one variable in the system.

Original: Let π G j denote the set of parents of node j in G and let X π G j denote the random vector containing the variables corresponding to the parents of j in G .
Revised: Let π G j denote the set of parents of node j in G and let X π G j denote the random vector containing the variables corresponding to the parents of j in G .

Original: Throughout the paper, we assume there are no hidden variables.
Revised: Throughout the paper, we assume there are no hidden variables.

Original: In a CGM, the distribution P X is said to be Markov to G , i.e. we can write the probability density function (pdf) of P X as p ( x ) =
Revised: In a CGM, the distribution P X is said to be Markov to G , i.e. we can write the probability density function (pdf) of P X as p ( x ) =

Original: (cid:81)
Revised: (cid:81)

Original: dj
Revised: dj

Original: =1 p j ( x j | x π G j ) where p j ( x j | x π G j ) is the conditional pdf of variable X j given X π G j .
Revised: =1 p j ( x j | x π G j ) where p j ( x j | x π G j ) is the conditional pdf of variable X j given X π G j .

Original: A CGM can be thought of as a BN in which directed edges are given a causal meaning, allowing it to answer queries regarding interventional distributions (Koller & Friedman, 2009).
Revised: A CGM can be thought of as a BN in which directed edges are given a causal meaning, allowing it to answer queries regarding interventional distributions (Koller & Friedman, 2009).

Original: 2. S TRUCTURE IDENTIFIABILITY
Revised: 2. S TRUCTURE IDENTIFIABILITY

Original: In general, it is impossible to recover G given only samples from P X and no intervention .
Revised: In general, it is impossible to recover G given only samples from P X , i.e. without interventional data .

Original: It is, however, customary to rely on a set of assumptions to render the structure fully or partially identiﬁable .
Revised: It is, however, customary to rely on a set of assumptions to render the structure fully or partially identiﬁable .

Original: Deﬁnition 

Revised: Deﬁnition

Original: Given a set of assumptions
Revised: Given a set of assumptions

Original: A on a CGM M =
Revised: A on a CGM M =

Original: ( P X , G ) , its graph G is said to be identiﬁable from P X if there exists
Revised: ( P X , G ) , its graph G is said to be identiﬁable from P X if there exists

Original: no other CGM ˜ M
Revised: no other CGM ˜ M

Original: =
Revised: =

Original: ( ˜ P X , ˜ G ) satisfying all assumptions in A such that ˜ G (cid:54) =
Revised: ( ˜ P X , ˜ G ) satisfying all assumptions in A such that ˜ G (cid:54) =

Original: G and ˜ P X
Revised: G and ˜ P X

Original: = P X .
Revised: = P X .

Original: There are many examples of graph identiﬁability results for continuous variables (Peters et al., 2014;
Revised: There are many examples of graph identiﬁability results for continuous variables (Peters et al., 2014;

Original: Peters & B¨uhlman, 2014; Shimizu et al., 2006; Zhang & Hyv¨arinen, 2009) as well as for discrete variables (Peters et al., 2011).
Revised: Peters & B¨uhlman, 2014; Shimizu et al., 2006; Zhang & Hyv¨arinen, 2009) as well as for discrete variables (Peters et al., 2011).

Original: These results are obtained by assuming that the conditional densities belong to a speciﬁc parametric family.
Revised: These results are obtained by assuming that the conditional densities belong to a speciﬁc parametric family.

Original: For example, if one assumes that the distribution P X is entailed by a structural equation model of the form
Revised: For example, if one assumes that the distribution P X is entailed by a structural equation model of the form

Original: [Equation]
Revised: [Equation]

Original: where f j is a nonlinear function satisfying some mild regularity conditions and the noises N j are mutually independent, then G is identiﬁable from P X (see Peters et al.
Revised: where f j is a nonlinear function satisfying some mild regularity conditions and the noises N j are mutually independent, then G is identiﬁable from P X (see Peters et al.

Original: (2014) for the complete theorem and its proof).
Revised: (2014) for the complete theorem and its proof).

Original: This is a particular instance of additive noise models (ANM).
Revised: This is a particular instance of additive noise models (ANM).

Original: We will make use of this result in our experiments in Section 4.
Revised: We will make use of this result in our experiments in Section 4.

Original: One can consider weaker assumptions such as faithfulness (Peters et al., 2017).
Revised: One can consider weaker assumptions such as faithfulness (Peters et al., 2017).

Original: This assumption allows one to identify, not G itself, but the Markov equivalence class to which it belongs (Spirtes et al., 2000).
Revised: This assumption allows one to identify, not G itself, but the Markov equivalence class to which it belongs (Spirtes et al., 2000).

Original: A Markov equivalence class is a set of DAGs which encodes exactly the same set of conditional independence statements and can be characterized by a graphical object named a completed partially directed acyclic graph (CPDAG) (Koller & Friedman, 2009; Peters et al., 2017).
Revised: A Markov equivalence class is a set of DAGs which encode exactly the same set of conditional independence statements and can be characterized by a graphical object named a completed partially directed acyclic graph (CPDAG) (Koller & Friedman, 2009; Peters et al., 2017).

Original: Some algorithms we use as baselines 
Revised: Some algorithms we use as baselines in Section 4 output only a CPDAG.

Original: 2. NOTEARS: C ONTINUOUS OPTIMIZATION FOR STRUCTURE LEARNING
Revised: 2. NOTEARS: C ONTINUOUS OPTIMIZATION FOR STRUCTURE LEARNING

Original: Structure learning is the problem of learning G using a data set of n samples { x (1) , ..., x ( n ) } from P X .
Revised: Structure learning is the problem of learning G using a data set of n samples { x (1) , ..., x ( n ) } from P X .

Original: Score-based approaches cast this problem as an optimization problem, i.e. ˆ
Revised: Score-based approaches cast this problem as an optimization problem, i.e. ˆ

Original: G = arg max G∈ DAG S ( G ) where S ( G ) is a regularized maximum likelihood under graph G .
Revised: G = arg max G∈ DAG S ( G ) where S ( G ) is a regularized maximum likelihood under graph G .

Original: Since the number of DAGs is super exponential in the number of nodes, most methods rely on various heuristic greedy search procedures to approximately solve the problem (see Section 5 for a review).
Revised: Since the number of DAGs is super exponential in the number of nodes, most methods rely on various heuristic greedy search procedures to approximately solve the problem (see Section 5 for a review).

Original: We now present the work of Zheng et al.
Revised: We now present the work of Zheng et al.

Original: (2018) which proposes to cast this combinatorial optimization problem into a continuous constrained one.
Revised: (2018) which proposes to cast this combinatorial optimization problem into a continuous constrained one.

Original: To do so, the authors propose to encode the graph G on d nodes as a weighted adjacency matrix
Revised: To do so, the authors propose to encode the graph G on d nodes as a weighted adjacency matrix

Original: U =
Revised: U =

Original: [ u 1 | . . .
Revised: [ u 1 | . . .

Original: | u d ] ∈ R d
Revised: | u d ] ∈ R d

Original: ×
Revised: ×

Original: d which represents (possibly negative) coefﬁcients in a linear SEM of the form X j := u (cid:62) j X + N i ∀ j where N j is a noise variable.
Revised: d which represents (possibly negative) coefﬁcients in a linear SEM of the form X j := u (cid:62) j X + N i ∀ j where N j is a noise variable.

Original: Let G U be the directed graph associated with the SEM and let A U be the (binary) adjacency matrix associated with G U .
Revised: Let G U be the directed graph associated with the SEM and let A U be the (binary) adjacency matrix associated with G U .

Original: One can see that the following equivalence holds:
Revised: One can see that the following equivalence holds:

Original: [Equation]
Revised: [Equation]

Original: To make sure G U is acyclic, the authors propose the following constraint on U :
Revised: To make sure G U is acyclic, the authors propose the following constraint on U :

Original: [Equation]
Revised: [Equation]

Original: where e M (cid:44) (cid:80) ∞
Revised: where e M (cid:44) (cid:80) ∞

Original: k =

Revised: k =

Original: M k k ! is the matrix exponential and (cid:12) is the Hadamard product.
Revised: M k k ! is the matrix exponential and (cid:12) is the Hadamard product.

Original: To see why this constraint characterizes acyclicity, ﬁrst note that ( A Uk )
Revised: To see why this constraint characterizes acyclicity, ﬁrst note that ( A Uk )

Original: jj is the number of cycles of length k passing through node j in graph G U .
Revised: jj is the number of cycles of length k passing through node j in graph G U .

Original: Clearly, for G U to be acyclic, we must have
Revised: Clearly, for G U to be acyclic, we must have

Original: Tr A Uk = 0 for k = 1 , 2 , ..., ∞ .
Revised: Tr A Uk = 0 for k = 1 , 2 , ..., ∞ .

Original: By equivalence (2), this is true when Tr( U (cid:12) U ) k
Revised: By equivalence (2), this is true when Tr( U (cid:12) U ) k

Original: = 0 for k = 1 , 2 , ..., ∞ .
Revised: = 0 for k = 1 , 2 , ..., ∞ .

Original: From there, one can simply apply the deﬁnition of the matrix exponential to see why constraint 3 characterizes acyclicity (see Zheng et al.
Revised: From there, one can simply apply the deﬁnition of the matrix exponential to see why constraint (3) characterizes acyclicity (see Zheng et al.

Original: (2018) for the full development).
Revised: (2018) for the full development).

Original: The authors propose to use a regularized negative least square score (maximum likelihood for a
Revised: The authors propose to use a regularized negative least square score (maximum likelihood for a

Original: Gaussian noise model).
Revised: Gaussian noise model).

Original: The resulting continuous constrained problem is
Revised: The resulting continuous constrained problem is

Original: [Equation]
Revised: [Equation]

Original: where X ∈ R n × d is the design matrix containing all n samples.
Revised: where X ∈ R n × d is the design matrix containing all n samples.

Original: The nature of the problem has been drastically changed: we went from a combinatorial to a continuous problem.
Revised: The nature of the problem has been drastically changed: we went from a combinatorial to a continuous problem.

Original: The difﬁculties of combinatorial optimization have been replaced by those of non-convex optimization, since the feasible set is non-convex.
Revised: The difﬁculties of combinatorial optimization have been replaced by those of non-convex optimization, since the feasible set is non-convex.

Original: Nevertheless, a standard numerical solver for constrained optimization such has an augmented Lagrangian method (Bertsekas, 1999) can be applied to get an approximate solution, hence there is no need to design a greedy search procedure.
Revised: Nevertheless, a standard numerical solver for constrained optimization such has an augmented Lagrangian method (Bertsekas, 1999) can be applied to get an approximate solution, hence there is no need to design a greedy search procedure.

Original: Moreover, this approach is more global than greedy methods in the sense that the whole matrix U is updated at each iteration.
Revised: Moreover, this approach is more global than greedy methods in the sense that the whole matrix U is updated at each iteration.

Original: Continuous approaches to combinatorial optimization have sometimes demonstrated improved performance over discrete approaches in the literature (see for example Alayrac et al.
Revised: Continuous approaches to combinatorial optimization have sometimes demonstrated improved performance over discrete approaches in the literature (see for example Alayrac et al.

Original: (2018, § 5.2) where they solve the multiple sequence alignment problem with a continuous optimization method).
Revised: (2018, § 5.2) where they solve the multiple sequence alignment problem with a continuous optimization method).

Original:  G RA N-DAG: G RADIENT - BASED NEURAL DAG LEARNING
Revised:  G RA N-DAG: G RADIENT - BASED NEURAL DAG LEARNING

Original: We propose a new nonlinear extension to the framework presented in Section 2.3.
Revised: We propose a new nonlinear extension to the framework presented in Section 2.3.

Original: For each variable X j , we learn a fully connected neural network with L hidden layers parametrized by φ ( j ) (cid:44) { W (1)( j ) , . . .
Revised: For each variable X j , we learn a fully connected neural network with L hidden layers parametrized by φ ( j ) (cid:44) { W (1)( j ) , . . .

Original: , W ( L +1)
Revised: , W ( L +1)

Original: ( j ) } where W ( (cid:96) )
Revised: ( j ) } where W ( (cid:96) )

Original: ( j ) is the (cid:96) th weight matrix of the j th NN (biases are omitted for clarity).
Revised: ( j ) is the (cid:96) th weight matrix of the j th NN (biases are omitted for clarity).

Original: Each NN takes as input X − j ∈ R d , i.e. the vector X with the j th component masked to zero, and outputs θ
Revised: Each NN takes as input X − j ∈ R d , i.e. the vector X with the j th component masked to zero, and outputs θ

Original: ( j ) ∈ R m , the m -dimensional parameter vector of the desired distribution family for variable X j . 

Revised: ( j ) ∈ R m , the m -dimensional parameter vector of the desired distribution family for variable X j .

Original: The fully connected NNs have the following form
Revised: The fully connected NNs have the following form

Original: [Equation]
Revised: [Equation]

Original: where g is a nonlinearity applied element-wise.
Revised: where g is a nonlinearity applied element-wise.

Original: Note that the evaluation of all NNs can be parallelized on GPU.
Revised: Note that the evaluation of all NNs can be parallelized on GPU.

Original: Distribution families need not be the same for each variable.
Revised: Distribution families need not be the same for each variable.

Original: Let φ (cid:44) [-
-]
Revised: Let φ (cid:44) [-
-]{ φ (1) , . .

Original: Not all parameter vectors need to have the same dimensionality, but to simplify the notation, we suppose m j = m ∀ j

Original: { φ (1) , . .

Original: , φ ( d ) } represents all parameters of all d NNs.
Revised: , φ ( d ) } represents all parameters of all d NNs.

Original: Without any constraint on its parameter φ ( j ) , neural network j models the conditional pdf p j ( x j | x − j ; φ ( j ) ) .
Revised: Without any constraint on its parameter φ ( j ) , neural network j models the conditional pdf p j ( x j | x − j ; φ ( j ) ) .

Original: Note that the product (cid:81) dj
Revised: Note that the product (cid:81) dj

Original: =1 p j ( x j | x − j ; φ ( j ) ) does not integrate to one (i.e. it is not a joint pdf), since it does not decompose according to a DAG.
Revised: =1 p j ( x j | x − j ; φ ( j ) ) does not integrate to one (i.e. it is not a joint pdf), since it does not decompose according to a DAG.

Original: We now show how one can constrain φ to make sure the product of all conditionals outputted by the NNs is a joint pdf.
Revised: We now show how one can constrain φ to make sure the product of all conditionals outputted by the NNs is a joint pdf.

Original: The idea is to deﬁne a new weighted adjacency matrix A φ similar to the one encountered in Section 2.3, which can be directly used inside the constraint of Equation 3 to enforce acyclicity.
Revised: The idea is to deﬁne a new weighted adjacency matrix A φ similar to the one encountered in Section 2.3, which can be directly used inside the constraint of Equation 3 to enforce acyclicity.

Original: 3. N EURAL NETWORK CONNECTIVITY
Revised: 3. N EURAL NETWORK CONNECTIVITY

Original: Before deﬁning the weighted adjacency matrix A φ , we need to focus on how one can make some
Revised: Before deﬁning the weighted adjacency matrix A φ , we need to focus on how one can make some

Original: NN outputs unaffected by some inputs.
Revised: NN outputs unaffected by some inputs.

Original: Since we will discuss properties of a single NN, we drop the
Revised: Since we will discuss properties of a single NN, we drop the

Original: NN subscript ( j ) to improve readability.
Revised: NN subscript ( j ) to improve readability.

Original: We will use the term neural network path to refer to a computation path in a NN.
Revised: We will use the term neural network path to refer to a computation path in a NN.

Original: For example, in a
Revised: For example, in a

Original: NN with two hidden layers, the sequence of weights ( W (1)
Revised: NN with two hidden layers, the sequence of weights ( W (1)

Original: h 1 i , W (2) h 2 h 1 , W (3) kh 2 ) is a NN path from input i to output k .
Revised: h 1 i , W (2) h 2 h 1 , W (3) kh 2 ) is a NN path from input i to output k .

Original: We say that a NN path is inactive if at least one weight along the path is zero.
Revised: We say that a NN path is inactive if at least one weight along the path is zero.

Original: We can loosely interpret the path product | W (1)
Revised: We can loosely interpret the path product | W (1)

Original: h 

Revised: h

Original: i || W (2) h 2 h 1 || W (3) kh 2 | ≥ 0 as the strength of the NN path, where a path product is equal to zero if and only if the path is inactive.
Revised: i || W (2) h 2 h 1 || W (3) kh 2 | ≥ 0 as the strength of the NN path, where a path product is equal to zero if and only if the path is inactive.

Original: Note that if all NN paths from input i to output k are inactive (i.e. the sum of their path products is zero), then output k does not depend on input i anymore since the information in input i will never reach output k .
Revised: Note that if all NN paths from input i to output k are inactive (i.e. the sum of their path products is zero), then output k does not depend on input i anymore since the information in input i will never reach output k .

Original: The sum of all path products from input i to output k for all input i and output k can be easily computed by taking the following matrix product.
Revised: The sum of all path products from input i to output k for all input i and output k can be easily computed by taking the following matrix product.

Original: [Equation]
Revised: [Equation]

Original: where | W | is the element-wise absolute value of W .
Revised: where | W | is the element-wise absolute value of W .

Original: Let us name C the neural network connectivity matrix .
Revised: Let us name C the neural network connectivity matrix .

Original: It can be veriﬁed that C ki is the sum of all NN path products from input i to output k .
Revised: It can be veriﬁed that C ki is the sum of all NN path products from input i to output k .

Original: This means it is sufﬁcient to have C ki = 

Revised: This means it is sufﬁcient to have C ki =

Original: to render output k independent of input i .
Revised: to render output k independent of input i .

Original: Remember that each NN in our model outputs a parameter vector θ for a conditional distribution and that we want the product of all conditionals to be a valid joint pdf, i.e. we want its corresponding directed graph to be acyclic.
Revised: Remember that each NN in our model outputs a parameter vector θ for a conditional distribution and that we want the product of all conditionals to be a valid joint pdf, i.e. we want its corresponding directed graph to be acyclic.

Original: With this in mind, we see that it could be useful to make a certain parameter θ not dependent on certain inputs of the NN.
Revised: With this in mind, we see that it could be useful to make a certain parameter θ not dependent on certain inputs of the NN.

Original: To have θ independent of variable X
Revised: To have θ independent of variable X

Original: i , it is sufﬁcient to have (cid:80) mk =1 C ki = 0 .
Revised: i , it is sufﬁcient to have (cid:80) mk =1 C ki = 0 .

Original: 3. A WEIGHTED ADJACENCY MATRIX
Revised: 3. A WEIGHTED ADJACENCY MATRIX

Original: We now deﬁne a weighted adjacency matrix A φ that can be used in constraint of Equation 3. 
Revised: We now deﬁne a weighted adjacency matrix A φ that can be used in constraint of Equation 3. [Equation]

Revised: otherwise

Original: [Equation]
Revised: [Equation]

Original: where C ( j ) denotes the connectivity matrix of the NN associated with variable X j .
Revised: where C ( j ) denotes the connectivity matrix of the NN associated with variable X j .

Original: As the notation suggests, A φ ∈ R d × d ≥ 0 depends on all weights of all NNs.
Revised: As the notation suggests, A φ ∈ R d × d ≥ 0 depends on all weights of all NNs.

Original: Moreover, it can effectively be interpreted as a weighted adjacency matrix similarly to what we presented in Section 2.3, since we have that
Revised: Moreover, it can effectively be interpreted as a weighted adjacency matrix similarly to what we presented in Section 2.3, since we have that

Original: [Equation]
Revised: [Equation]

Original: We note G φ to be the directed graph entailed by parameter φ .
Revised: We note G φ to be the directed graph entailed by parameter φ .

Original: We can now write our adapted acyclicity constraint:
Revised: We can now write our adapted acyclicity constraint:

Original: [Equation]
Revised: [Equation]

Original: Note that we can compute the gradient of h ( φ ) w.r.t.
Revised: Note that we can compute the gradient of h ( φ ) w.r.t.

Original: φ (except at points of non-differentiability arising from the absolute value function, similar to standard neural networks with ReLU activations (Glorot et al., 2011); these points did not appear problematic in our experiments using SGD).
Revised: φ (except at points of non-differentiability arising from the absolute value function, similar to standard neural networks with ReLU activations (Glorot et al., 2011); these points did not appear problematic in our experiments using SGD).

Original: 3. A DIFFERENTIABLE SCORE AND ITS OPTIMIZATION
Revised: 3. A DIFFERENTIABLE SCORE AND ITS OPTIMIZATION

Original: We propose solving the maximum likelihood optimization problem
Revised: We propose solving the maximum likelihood optimization problem

Original: [Equation]
Revised: [Equation]

Original: where π φj denotes the set of parents of node j in graph G φ .
Revised: where π φj denotes the set of parents of node j in graph G φ .

Original: Note that (cid:80) dj
Revised: Note that (cid:80) dj

Original: =1 log p j
Revised: =1 log p j

Original: ( X j | X π φj ; φ ( j ) ) is a valid log-likelihood function when constraint (9) is satisﬁed.
Revised: ( X j | X π φj ; φ ( j ) ) is a valid log-likelihood function when constraint (9) is satisﬁed.

Original: As suggested in Zheng et al.
Revised: As suggested in Zheng et al.

Original: (2018), we apply an augmented Lagrangian approach to get an approximate solution to program (10).
Revised: (2018), we apply an augmented Lagrangian approach to get an approximate solution to program (10).

Original: Augmented Lagrangian methods consist of optimizing a sequence of subproblems for which the exact solutions are known to converge to a stationary point of the constrained problem under some regularity conditions (Bertsekas, 1999).
Revised: Augmented Lagrangian methods consist of optimizing a sequence of subproblems for which the exact solutions are known to converge to a stationary point of the constrained problem under some regularity conditions (Bertsekas, 1999).

Original: In our case, each subproblem is
Revised: In our case, each subproblem is

Original: [Equation]
Revised: [Equation]

Original: where λ t and µ t are the Lagrangian and penalty coefﬁcients of the t th subproblem, respectively.
Revised: where λ t and µ t are the Lagrangian and penalty coefﬁcients of the t th subproblem, respectively.

Original: These coefﬁcients are updated after each subproblem is solved.
Revised: These coefﬁcients are updated after each subproblem is solved.

Original: Since GraN-DAG rests on neural networks, we propose to approximately solve each subproblem using a well-known stochastic gradient algorithm popular for NN in part for its implicit regularizing effect (Poggio et al., 2018).
Revised: Since GraN-DAG rests on neural networks, we propose to approximately solve each subproblem using a well-known stochastic gradient algorithm popular for NN in part for its implicit regularizing effect (Poggio et al., 2018).

Original: See
Revised: See

Original: Appendix A.1 for details regarding the optimization procedure.
Revised: Appendix A.1 for details regarding the optimization procedure.

Original: In the current section, we presented GraN-DAG in a general manner without specifying explicitly which distribution family is parameterized by θ ( j ) .
Revised: In the current section, we presented GraN-DAG in a general manner without specifying explicitly which distribution family is parameterized by θ ( j ) .

Original: In principle, any distribution family could be employed as long as its log-likelihood can be computed and differentiated with respect to its parameter θ .
Revised: In principle, any distribution family could be employed as long as its log-likelihood can be computed and differentiated with respect to its parameter θ .

Original: However, it is not always clear whether the exact solution of problem (10) recovers the ground truth graph G .
Revised: However, it is not always clear whether the exact solution of problem (10) recovers the ground truth graph G .

Original: It will depend on both the modelling choice of GraN-DAG and the underlying CGM ( P X , G ) .
Revised: It will depend on both the modelling choice of GraN-DAG and the underlying CGM ( P X , G ) .

Original: Proposition 1 Let φ ∗ and G φ ∗ be the optimal solution to (10) and its corresponding graph, respectively.
Revised: Proposition 1 Let φ ∗ and G φ ∗ be the optimal solution to (10) and its corresponding graph, respectively.

Original: Let M ( A ) be the set of CGM ( P (cid:48) , G (cid:48) ) for which the assumptions in A are satisﬁed and let C be the set of CGM ( P (cid:48) , G (cid:48) ) which can be represented by the model (e.g. NN outputting a
Revised: Let M ( A ) be the set of CGM ( P (cid:48) , G (cid:48) ) for which the assumptions in A are satisﬁed and let C be the set of CGM ( P (cid:48) , G (cid:48) ) which can be represented by the model (e.g. NN outputting a

Original: Gaussian distribution).
Revised: Gaussian distribution).

Original: If the underlying CGM ( P X , G ) ∈ C and C = M ( A ) for a speciﬁc set of assumptions A such that G is identiﬁable from P X , then G φ ∗ = G .
Revised: If the underlying CGM ( P X , G ) ∈ C and C = M ( A ) for a speciﬁc set of assumptions A such that G is identiﬁable from P X , then G φ ∗ = G .

Original: Proof: Let P φ be the joint distribution entailed by parameter φ .
Revised: Proof: Let P φ be the joint distribution entailed by parameter φ .

Original: Note that the population loglikelihood E X ∼ P
Revised: Note that the population loglikelihood E X ∼ P

Original: X log p φ ( X ) is maximal iff P φ
Revised: X log p φ ( X ) is maximal iff P φ

Original: = P X .
Revised: = P X .

Original: We know this maximum can be achieved by a speciﬁc parameter φ ∗ since by hypothesis ( P X , G ) ∈ C .
Revised: We know this maximum can be achieved by a speciﬁc parameter φ ∗ since by hypothesis ( P X , G ) ∈ C .

Original: Since G is identiﬁable from P X , we know there exists no other CGM ( ˜ P X , ˜ G
Revised: Since G is identiﬁable from P X , we know there exists no other CGM ( ˜ P X , ˜ G

Original: ) ∈ C such that ˜ G (cid:54) =
Revised: ) ∈ C such that ˜ G (cid:54) =

Original: G and ˜ P X
Revised: G and ˜ P X

Original: = P X .
Revised: = P X .

Original: Hence G φ ∗ has to be equal to G .
Revised: Hence G φ ∗ has to be equal to G .

Original: (cid:4)
Revised: (cid:4)

Original: In Section 4.1, we empirically explore the identiﬁable setting of nonlinear Gaussian ANMs introduced in Section 2.2.
Revised: In Section 4.1, we empirically explore the identiﬁable setting of nonlinear Gaussian ANMs introduced in Section 2.2.

Original: In practice, one should keep in mind that solving (10) exactly is hard since the problem is non-convex (the augmented Lagrangian converges only to a stationary point) and moreover we only have access to the empirical log-likelihood (Proposition 1 holds for the population case).
Revised: In practice, one should keep in mind that solving (10) exactly is hard since the problem is non-convex (the augmented Lagrangian converges only to a stationary point) and moreover we only have access to the empirical log-likelihood (Proposition 1 holds for the population case).

Original: 3. T HRESHOLDINGTO ENSURE ACYCLICITY
Revised: 3. T HRESHOLDING

Original: The solution outputted by the augmented Lagrangian will satisfy the constraint only up to numerical precision, thus several entries of A φ might not be exactly zero and require thresholding.
Revised: The solution outputted by the augmented Lagrangian will satisfy the constraint only up to numerical precision, thus several entries of A φ might not be exactly zero and require thresholding.

Original: To do so, we mask the inputs of each NN j using a binary matrix M ( j ) ∈ { 0 , 1 }
Revised: To do so, we mask the inputs of each NN j using a binary matrix M ( j ) ∈ { 0 , 1 }

Original: d × d initialized to have ( M ( j ) ) ii = 1 ∀
Revised: d × d initialized to have ( M ( j ) ) ii = 1 ∀

Original: i (cid:54) =
Revised: i (cid:54) =

Original: j and zeros everywhere else.
Revised: j and zeros everywhere else.

Original: Having ( M ( j ) ) ii = 0 means the input i of NN j has been thresholded.
Revised: Having ( M ( j ) ) ii = 0 means the input i of NN j has been thresholded.

Original: This mask is integrated in the product of Equation 6 by doing C ( j ) (cid:44)
Revised: This mask is integrated in the product of Equation 6 by doing C ( j ) (cid:44)

Original: | W ( L +1)
Revised: | W ( L +1)

Original: ( j ) | . . .
Revised: ( j ) | . . .

Original: | W (1)( j ) | M ( j ) without changing the interpretation of C ( j ) .
Revised: | W (1)( j ) | M ( j ) without changing the interpretation of C ( j ) ( M ( j ) can be seen simply as an extra layer in the NN).

Original: During optimization, if the entry ( A φ ) ij is smaller than the threshold (cid:15) =
Revised: During optimization, if the entry ( A φ ) ij is smaller than the threshold (cid:15) =

Original: 10 − 4 , the corresponding mask entry ( M ( j ) ) ii is set to zero, permanently.
Revised: 10 − 4 , the corresponding mask entry ( M ( j ) ) ii is set to zero, permanently.

Original: The masks M ( j ) are never updated via gradient descent.
Revised: The masks M ( j ) are never updated via gradient descent.

Original: We also add an iterative thresholding step at the end to ensure the estimated graph G φ is acyclic (described in Appendix A.2).
Revised: We also add an iterative thresholding step at the end to ensure the estimated graph G φ is acyclic (described in Appendix A.2).

Original: 3. O VERFITTING
Revised: 3. O VERFITTING

Original: In practice, we maximize an empirical estimate of the objective of problem (10).
Revised: In practice, we maximize an empirical estimate of the objective of problem (10).

Original: It is well known that this maximum likelihood score is prone to overﬁtting in the sense that adding edges can never reduce the maximal likelihood (Koller & Friedman, 2009).
Revised: It is well known that this maximum likelihood score is prone to overﬁtting in the sense that adding edges can never reduce the maximal likelihood (Koller & Friedman, 2009).

Original: GraN-DAG gets around this issue in four ways.
Revised: GraN-DAG gets around this issue in four ways.

Original: First, as we optimize a subproblem, we evaluate its objective on a held-out data set and declare convergence once it has stopped improving.
Revised: First, as we optimize a subproblem, we evaluate its objective on a held-out data set and declare convergence once it has stopped improving.

Original: This approach is known as early stopping (Prechelt, 1997).
Revised: This approach is known as early stopping (Prechelt, 1997).

Original: Second, to optimize (11), we use a stochastic gradient algorithm variant which is now known to have an implicit regularizing effect (Poggio et al., 2018).
Revised: Second, to optimize (11), we use a stochastic gradient algorithm variant which is now known to have an implicit regularizing effect (Poggio et al., 2018).

Original: Third, once we have thresholded our graph estimate to be a DAG, we apply a ﬁnal pruning step identical to what is done in CAM (B¨uhlmann et al., 2014) to remove spurious edges.
Revised: Third, once we have thresholded our graph estimate to be a DAG, we apply a ﬁnal pruning step identical to what is done in CAM (B ¨ uhlmann et al., 2014) to remove spurious edges.

Original: This step performs a regression of each node against its parents and uses a signiﬁcance test to decide which parents should be kept or not.
Revised: This step performs a regression of each node against its parents and uses a signiﬁcance test to decide which parents should be kept or not.

Original: Fourth, for graphs of 50 nodes or more, we apply a preliminary neighbors selection (PNS) before running the optimization procedure as was also recommended in B¨uhlmann et al.
Revised: Fourth, for graphs of 50 nodes or more, we apply a preliminary neighbors selection (PNS) before running the optimization procedure as was also recommended in B ¨ uhlmann et al.

Original: This procedure selects a set of potential parents for each variables.
Revised: This procedure selects a set of potential parents for each variables.

Original: See Appendix A.3 for details on PNS and pruning.
Revised: See Appendix A.3 for details on PNS and pruning.

Original: Many score-based approaches control overﬁtting by penalizing the number of edges in their score.
Revised: Many score-based approaches control overﬁtting by penalizing the number of edges in their score.

Original: For example, NOTEARS includes the L1 norm of its weighted adjacency matrix U in its objective.
Revised: For example, NOTEARS includes the L1 norm of its weighted adjacency matrix U in its objective.

Original: GraN-DAG regularizes using PNS and pruning for ease of comparision to CAM, the most competitive approach in our experiments.
Revised: GraN-DAG regularizes using PNS and pruning for ease of comparision to CAM, the most competitive approach in our experiments.

Original: The importance of PNS and pruning and their ability to reduce overﬁtting is illustrated in an ablation study presented in Appendix A.3.
Revised: The importance of PNS and pruning and their ability to reduce overﬁtting is illustrated in an ablation study presented in Appendix A.3.

Original: The study shows that PNS and pruning are both very important for the performance of GraN-DAG in terms of SHD, but do not have a signiﬁcant effect in terms of SID.
Revised: The study shows that PNS and pruning are both very important for the performance of GraN-DAG in terms of SHD, but do not have a signiﬁcant effect in terms of SID.

Original: In these experiments, we also present
Revised: In these experiments, we also present

Original: NOTEARS and DAG-GNN with PNS and pruning, without noting a signiﬁcant improvement.
Revised: NOTEARS and DAG-GNN with PNS and pruning, without noting a signiﬁcant improvement.

Original: 3. C OMPUTATIONAL C OMPLEXITY
Revised: 3. C OMPUTATIONAL C OMPLEXITY

Original: To learn a graph, GraN-DAG relies on the proper training of neural networks on which it is built.
Revised: To learn a graph, GraN-DAG relies on the proper training of neural networks on which it is built.

Original: We thus propose using a stochastic gradient method which is a standard choice when it comes to NN training because it scales well with both the sample size and the number of parameters and it implicitly regularizes learning.
Revised: We thus propose using a stochastic gradient method which is a standard choice when it comes to NN training because it scales well with both the sample size and the number of parameters and it implicitly regularizes learning.

Original: Similarly to NOTEARS, GraN-DAG requires the evaluation of the matrix exponential of A φ at each iteration costing O ( d 3 ) .
Revised: Similarly to NOTEARS, GraN-DAG requires the evaluation of the matrix exponential of A φ at each iteration costing O ( d 3 ) .

Original: NOTEARS justiﬁes the use of a batch proximal quasi-Newton algorithm by the low number of O ( d 3 ) iterations required to converge.
Revised: NOTEARS justiﬁes the use of a batch proximal quasi-Newton algorithm by the low number of O ( d 3 ) iterations required to converge.

Original: Since GraNDAG uses a stochastic gradient method, one would expect it will require more iterations to converge.
Revised: Since GraNDAG uses a stochastic gradient method, one would expect it will require more iterations to converge.

Original: However, in practice we observe that GraN-DAG performs fewer iteration than NOTEARS before the augmented Lagrangian converges (see Table 4 of Appendix A.1).
Revised: However, in practice we observe that GraN-DAG performs fewer iterations than NOTEARS before the augmented Lagrangian converges (see Table 4 of Appendix A.1).

Original: We hypothesize this is due to early stopping which avoids having to wait until the full convergence of the subproblems hence limiting the total number of iterations.
Revised: We hypothesize this is due to early stopping which avoids having to wait until the full convergence of the subproblems hence limiting the total number of iterations.

Original: Moreover, for the graph sizes considered in this paper ( d ≤ 100 ), the evaluation of h ( φ ) in GraN-DAG, which includes the matrix exponentiation, does not dominate the cost of each iteration ( ≈ 4% for 20 nodes and ≈ 13% for 100 nodes graphs).
Revised: Moreover, for the graph sizes considered in this paper ( d ≤ 100 ), the evaluation of h ( φ ) in GraN-DAG, which includes the matrix exponentiation, does not dominate the cost of each iteration ( ≈ 4% for 20 nodes and ≈ 13% for 100 nodes graphs).

Original: Evaluating the approximate gradient of the log-likelihood (costing O ( d 2 ) assuming a ﬁxed minibatch size and NN depth and width) appears to be of greater importance for d ≤ 100 .
Revised: Evaluating the approximate gradient of the log-likelihood (costing O ( d 2 ) assuming a ﬁxed minibatch size, NN depth and width) appears to be of greater importance for d ≤ 100 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we compare GraN-DAG to various baselines in the continuous paradigm, namely
Revised: In this section, we compare GraN-DAG to various baselines in the continuous paradigm, namely

Original: DAG-GNN (Yu et al., 2019) and NOTEARS (Zheng et al., 2018), and also in the combinatorial paradigm, namely CAM (B¨uhlmann et al., 2014), GSF (Huang et al., 2018), GES (Chickering, 2003) and PC (Spirtes
Revised: DAG-GNN (Yu et al., 2019) and NOTEARS (Zheng et al., 2018), and also in the combinatorial paradigm, namely CAM (B¨uhlmann et al., 2014), GSF (Huang et al., 2018), GES (Chickering, 2003) and PC (Spirtes

Original: et al., 2000).
Revised: et al., 2000).

Original: These methods are discussed in Section 5.
Revised: These methods are discussed in Section 5.

Original: In all experiments, each NN learned by GraN-DAG outputs the mean of a Gaussian distribution ˆ µ ( j ) , i.e. θ ( j ) :
Revised: In all experiments, each NN learned by GraN-DAG outputs the mean of a Gaussian distribution ˆ µ ( j ) , i.e. θ ( j ) :

Original: = ˆ µ ( j ) and X j | X π G j ∼ N (ˆ µ ( j ) , ˆ σ 2( j ) ) ∀ j .
Revised: = ˆ µ ( j ) and X j | X π G j ∼ N (ˆ µ ( j ) , ˆ σ 2 ( j ) ) ∀ j .

Original: The parameters ˆ σ 2( j ) are learned as well, but do not depend on the parent variables
Revised: The parameters ˆ σ 2 ( j ) are learned as well, but do not depend on the parent variables

Original: X π G j (unless otherwise stated).
Revised: X π G j (unless otherwise stated).

Original: Note that this modelling choice matches the nonlinear Gaussian ANM introduced in Section 2.2.
Revised: Note that this modelling choice matches the nonlinear Gaussian ANM introduced in Section 2.2.

Original: We report the performance of random graphs sampled using the Erd˝os - R´enyi (ER) scheme described in Appendix A.5 (denoted by RANDOM).
Revised: We report the performance of random graphs sampled using the Erd˝os - R´enyi (ER) scheme described in Appendix A.5 (denoted by RANDOM).

Original: For each approach, we evaluate the estimated graph on two metrics: the structural hamming distance (SHD) and the structural interventional distance (SID) (Peters & B¨uhlmann, 2013).
Revised: For each approach, we evaluate the estimated graph on two metrics: the structural hamming distance (SHD) and the structural interventional distance (SID) (Peters & B¨uhlmann, 2013).

Original: The former simply counts the number of missing, falsely detected or reversed edges.
Revised: The former simply counts the number of missing, falsely detected or reversed edges.

Original: The latter is especially well suited for causal inference since it counts the number of couples ( i, j ) such that the interventional distribution p ( x j | do ( X i = ¯ x )) would be miscalculated if we were to use the estimated graph to form the parent adjustement set.
Revised: The latter is especially well suited for causal inference since it counts the number of couples ( i, j ) such that the interventional distribution p ( x j | do ( X i = ¯ x )) would be miscalculated if we were to use the estimated graph to form the parent adjustement set.

Original: Note that
Revised: Note that

Original: GSF, GES and PC output only a CPDAG, hence the need to report a lower and an upper bound on the
Revised: GSF, GES and PC output only a CPDAG, hence the need to report a lower and an upper bound on the

Original: SID.
Revised: SID.

Original: See Appendix A.7 for more details on SHD and SID.
Revised: See Appendix A.7 for more details on SHD and SID.

Original: All experiments were ran with publicly available code from the authors website.
Revised: All experiments were ran with publicly available code from the authors website.

Original: See Appendix A.8 for the details of their hyperparameters.
Revised: See Appendix A.8 for the details of their hyperparameters.

Original: In Appendix A.9, we explain how one could use a held-out data set to select the hyperparameters of score-based approaches and report the results of such a procedure on almost every settings discussed in the present section.
Revised: In Appendix A.9, we explain how one could use a held-out data set to select the hyperparameters of score-based approaches and report the results of such a procedure on almost every settings discussed in the present section.

Original: 4. S YNTHETIC DATA
Revised: 4. S YNTHETIC DATA

Original: We have generated different data set types which vary along four dimensions: data generating process, number of nodes, level of edge sparsity and graph type.
Revised: We have generated different data set types which vary along four dimensions: data generating process, number of nodes, level of edge sparsity and graph type.

Original: We consider two graph sampling schemes: Erd˝os - R´enyi (ER) and scale-free (SF) (see Appendix A.5 for details).
Revised: We consider two graph sampling schemes: Erd˝os - R´enyi (ER) and scale-free (SF) (see Appendix A.5 for details).

Original: For each data set type, we sampled 10 data sets of 1000 examples as follows:
Revised: For each data set type, we sampled 10 data sets of 1000 examples as follows:

Original: First, a ground truth DAG G is randomly sampled following either the ER or the SF scheme.
Revised: First, a ground truth DAG G is randomly sampled following either the ER or the SF scheme.

Original: Then, the data is generated according to a speciﬁc sampling scheme.
Revised: Then, the data is generated according to a speciﬁc sampling scheme.

Original: The ﬁrst data generating process we consider is the nonlinear Gaussian ANM (Gauss-ANM) introduced in Section 2.2 in which data is sampled following X j :=
Revised: The ﬁrst data generating process we consider is the nonlinear Gaussian ANM (Gauss-ANM) introduced in Section 2.2 in which data is sampled following X j :=

Original: f j ( X π G j ) + N j with mutually independent noises N j ∼ N (0 , σ 2 j ) ∀ j where the functions f
Revised: f j ( X π G j ) + N j with mutually independent noises N j ∼ N (0 , σ 2 j ) ∀ j where the functions f

Original: j are independently sampled from a
Revised: j are independently sampled from a

Original: Gaussian process with a unit bandwidth RBF kernel and with σ 2 j sampled uniformly.
Revised: Gaussian process with a unit bandwidth RBF kernel and with σ 2 j sampled uniformly.

Original: As mentioned in Section 2.2, we know G to be identiﬁable from the distribution.
Revised: As mentioned in Section 2.2, we know G to be identiﬁable from the distribution.

Original: Proposition 1 indicates that the modelling choice of GraN-DAG together with this synthetic data ensure that solving (10) to optimality would recover the correct graph.
Revised: Proposition 1 indicates that the modelling choice of GraN-DAG together with this synthetic data ensure that solving (10) to optimality would recover the correct graph.

Original: Note that NOTEARS and CAM also make the correct Gaussian noise assumption, but do not have enough capacity to represent the f j functions properly.
Revised: Note that NOTEARS and CAM also make the correct Gaussian noise assumption, but do not have enough capacity to represent the f j functions properly.

Original: We considered graphs of 10, 20, 50 and 100 nodes.
Revised: We considered graphs of 10, 20, 50 and 100 nodes.

Original: Tables 1 & 2 present results only for 10 and 50 nodes since the conclusions do not change with graphs of 20 or 100 nodes (see Appendix A. for these additional experiments).
Revised: Tables 1 & 2 present results only for 10 and 50 nodes since the conclusions do not change with graphs of 20 or 100 nodes (see Appendix A. for these additional experiments).

Original: We consider graphs of d and 4 d edges (respectively denoted by
Revised: We consider graphs of d and 4 d edges (respectively denoted by

Original: ER1 and ER4 in the case of ER graphs).
Revised: ER1 and ER4 in the case of ER graphs).

Original: We report the performance of the popular GES and PC in
Revised: We report the performance of the popular GES and PC in

Original: Appendix A.6 since they are almost never on par with the best methods presented in this section.
Revised: Appendix A.6 since they are almost never on par with the best methods presented in this section.

Original: [Table]
Revised: [Table]

Revised: NOTEARS

Original: [Table]
Revised: [Table]

Original:  We now examine Tables 1 & 2 (the errors bars represent the standard deviation across datasets per task).
Revised: [Table] We now examine Tables 1 & 2 (the errors bars represent the standard deviation across datasets per task).

Original: We can see that, across all settings, GraN-DAG and CAM are the best performing methods, both in terms of SHD and SID, while GSF is not too far behind.
Revised: We can see that, across all settings, GraN-DAG and CAM are the best performing methods, both in terms of SHD and SID, while GSF is not too far behind.

Original: The poor performance of NOTEARS can be explained by its inability to model nonlinear functions.
Revised: The poor performance of NOTEARS can be explained by its inability to model nonlinear functions.

Original: In terms of SHD, DAG-GNN performs rarely better than NOTEARS while in terms of SID, it performs similarly to RANDOM in almost all cases except in scale-free networks of 50 nodes or more.
Revised: In terms of SHD, DAG-GNN performs rarely better than NOTEARS while in terms of SID, it performs similarly to RANDOM in almost all cases except in scale-free networks of 50 nodes or more.

Original: Its poor performance might be due to its incorrect modelling assumptions and because its architecture uses a strong form of parameter sharing between the f j functions, which is not justiﬁed in a setup like ours.
Revised: Its poor performance might be due to its incorrect modelling assumptions and because its architecture uses a strong form of parameter sharing between the f j functions, which is not justiﬁed in a setup like ours.

Original: GSF performs always better than DAG-GNN and NOTEARS but performs as good as CAM and GraN-DAG only about half the time.
Revised: GSF performs always better than DAG-GNN and NOTEARS but performs as good as CAM and GraN-DAG only about half the time.

Original: Among the continuous approaches considered, GraN-DAG is the best performing on these synthetic tasks.
Revised: Among the continuous approaches considered, GraN-DAG is the best performing on these synthetic tasks.

Original: Even though
Revised: Even though

Original: CAM (wrongly) assumes that the functions
Revised: CAM (wrongly) assumes that the functions

Original: f j are additive, i.e. f j ( x π G j )
Revised: f j are additive, i.e. f j ( x π G j )

Original: = (cid:80) i ∈
Revised: = (cid:80) i ∈

Original: π G j f ij ( x j ) ∀
Revised: π G j f ij ( x j ) ∀

Original: j , it manages to compete with GraN-DAG which does not make this incorrect modelling assumption 3 .
Revised: j , it manages to compete with GraN-DAG which does not make this incorrect modelling assumption 2 .

Original: This might partly be explained by a bias-variance trade-off.
Revised: This might partly be explained by a bias-variance trade-off.

Original: CAM is biased but has a lower variance than GraN-DAG due to its restricted capacity, resulting in both methods performing similarly.
Revised: CAM is biased but has a lower variance than GraN-DAG due to its restricted capacity, resulting in both methods performing similarly.

Original: In Appendix A.4, we present an experiment showing that
Revised: In Appendix A.4, we present an experiment showing that

Original: GraN-DAG can outperform CAM in higher sample size settings, suggesting this explanation is reasonable.
Revised: GraN-DAG can outperform CAM in higher sample size settings, suggesting this explanation is reasonable.

Original: Having conﬁrmed that GraN-DAG is competitive on the ideal Gauss-ANM data, we experimented with settings better adjusted to other models to see whether GraN-DAG remains competitive.
Revised: Having conﬁrmed that GraN-DAG is competitive on the ideal Gauss-ANM data, we experimented with settings better adjusted to other models to see whether GraN-DAG remains competitive.

Original: We considered linear Gaussian data (better adjusted to NOTEARS) and nonlinear Gaussian data with additive functions (better adjusted to CAM) named LIN and ADD-FUNC, respectively.
Revised: We considered linear Gaussian data (better adjusted to NOTEARS) and nonlinear Gaussian data with additive functions (better adjusted to CAM) named LIN and ADD-FUNC, respectively.

Original: See Appendix A.5 for the details of their generation.
Revised: See Appendix A.5 for the details of their generation.

Original: We report the results of GraN-DAG and other baselines in Table 12 & 13 of the appendix.
Revised: We report the results of GraN-DAG and other baselines in Table 12 & 13 of the appendix.

Original: On linear Gaussian data, most methods score poorly in terms of SID which is probably due to the unidentiﬁability of the linear Gaussian model (when the noise variances are unequal).
Revised: On linear Gaussian data, most methods score poorly in terms of SID which is probably due to the unidentiﬁability of the linear Gaussian model (when the noise variances are unequal).

Original: GraN-DAG and CAM perform similarly to NOTEARS in terms of SHD.
Revised: GraN-DAG and CAM perform similarly to NOTEARS in terms of SHD.

Original: On
Revised: On

Original: ADD-FUNC, CAM dominates all methods on most graph types considered (GraN-DAG is on par only for the 10 nodes ER1 graph).
Revised: ADD-FUNC, CAM dominates all methods on most graph types considered (GraN-DAG is on par only for the 10 nodes ER1 graph).

Original: However, GraN-DAG outperforms all other methods which can be explained by the fact that the conditions of Proposition 1 are satisﬁed (supposing the functions (cid:80) i ∈
Revised: However, GraN-DAG outperforms all other methods which can be explained by the fact that the conditions of Proposition 1 are satisﬁed (supposing the functions (cid:80) i ∈

Original: π G j f ij ( X i ) can be represented by the NNs).
Revised: π G j f ij ( X i ) can be represented by the NNs).

Original: We also considered synthetic data sets which do not satisfy the additive Gaussian noise assumption present in GraN-DAG, NOTEARS and CAM.
Revised: We also considered synthetic data sets which do not satisfy the additive Gaussian noise assumption present in GraN-DAG, NOTEARS and CAM.

Original: We considered two kinds of post nonlinear causal models (Zhang & Hyv¨arinen, 2009), PNL-GP and PNL-MULT (see Appendix A.5 for details about their generation).
Revised: We considered two kinds of post nonlinear causal models (Zhang & Hyv ¨ arinen, 2009), PNL-GP and PNL-MULT (see Appendix A.5 for details about their generation).

Original: A post nonlinear model has the form X j :=
Revised: A post nonlinear model has the form X j :=

Original: g j ( f j ( X π G j ) + N j ) where N j is a noise variable.
Revised: g j ( f j ( X π G j ) + N j ) where N j is a noise variable.

Original: Note that GraN-DAG (with the current modelling choice) and CAM do not have the representational power to express these conditional distributions, hence violating an assumption of Proposition 1.
Revised: Note that GraN-DAG (with the current modelling choice) and CAM do not have the representational power to express these conditional distributions, hence violating an assumption of Proposition 1.

Original: However, these data sets differ from the previous additive noise setup only by the nonlinearity g j , hence offering a case of mild model misspeciﬁcation.
Revised: However, these data sets differ from the previous additive noise setup only by the nonlinearity g j , hence offering a case of mild model misspeciﬁcation.

Original: The results are reported in Table 14 of the appendix.
Revised: The results are reported in Table 14 of the appendix.

Original: GraN-DAG and CAM are outperforming DAG-GNN and NOTEARS except in SID for certain data sets where all methods score similarly to RANDOM.
Revised: GraN-DAG and CAM are outperforming DAG-GNN and NOTEARS except in SID for certain data sets where all methods score similarly to RANDOM.

Original: GraN-DAG and CAM have similar performance on all data sets except one where CAM is better.
Revised: GraN-DAG and CAM have similar performance on all data sets except one where CAM is better.

Original: GSF performs worst than GraN-DAG (in both SHD and SID) on PNL-GP but not on PNL-MULT where it performs better in SID.
Revised: GSF performs worst than GraN-DAG (in both SHD and SID) on PNL-GP but not on PNL-MULT where it performs better in SID.

Original: 4. R EAL AND PSEUDO - REAL DATA
Revised: 4. R EAL AND PSEUDO - REAL DATA

Original: We have tested all methods considered so far on a well known data set which measures the expression level of different proteins and phospholipids in human cells (Sachs et al., 2005).
Revised: We have tested all methods considered so far on a well known data set which measures the expression level of different proteins and phospholipids in human cells (Sachs et al., 2005).

Original: We trained only on the n = 853 observational samples.
Revised: We trained only on the n = 853 observational samples.

Original: This dataset and its ground truth graph proposed in Sachs et al.
Revised: This dataset and its ground truth graph proposed in Sachs et al.

Original: (2005) (11 nodes and 17 edges) are often used in the probabilistic graphical model literature (Koller & Friedman, 2009).
Revised: (2005) (11 nodes and 17 edges) are often used in the probabilistic graphical model literature (Koller & Friedman, 2009).

Original: We also consider pseudo-real data sets sampled from the SynTReN generator (Van den Bulcke, 2006).
Revised: We also consider pseudo-real data sets sampled from the SynTReN generator (Van den Bulcke, 2006).

Original: This generator was designed to create synthetic transcriptional regulatory networks and produces simulated gene expression data that approximates experimental data.
Revised: This generator was designed to create synthetic transcriptional regulatory networks and produces simulated gene expression data that approximates experimental data.

Original: See
Revised: See

Original: Appendix A.5 for details of the generation.
Revised: Appendix A.5 for details of the generation.

Original: In applications, it is not clear whether the conditions of Proposition 1 hold since we do not know whether ( P X , G ) ∈ C .
Revised: In applications, it is not clear whether the conditions of Proposition 1 hold since we do not know whether ( P X , G ) ∈ C .

Original: This departure from identiﬁable settings is an occasion to explore a different modelling choice for GraN-DAG.
Revised: This departure from identiﬁable settings is an occasion to explore a different modelling choice for GraN-DAG.

Original: In addition to the model presented at the beginning of this section, we consider an alternative, denoted GraN-DAG++, which allows the variance parameters ˆ σ 2 ( i ) to depend on the parent variables
Revised: In addition to the model presented at the beginning of this section, we consider an alternative, denoted GraN-DAG++, which allows the variance parameters ˆ σ 2( i ) to depend on the parent variables

Original: X π G i through the NN, i.e. θ ( i ) :
Revised: X π G i through the NN, i.e. θ ( i ) :

Original: = (ˆ µ ( i ) , log ˆ σ 2( i ) ) .
Revised: = (ˆ µ ( i ) , log ˆ σ 2( i ) ) .

Original: Note that this is violating the additive noise assumption (in ANMs, the noise is independent of the parent variables).
Revised: Note that this is violating the additive noise assumption (in ANMs, the noise is independent of the parent variables).

Original: In addition to metrics used in Section 4.
Revised: In addition to metrics used in Section 4.

Original: [Table]
Revised: [Table]

Original: we also report SHD-C. To compute the SHD-C between two DAGs [Table] we ﬁrst map each of them to their corresponding CPDAG and measure the SHD between the two.
Revised: we also report SHD-C. To compute the SHD-C between two DAGs, we ﬁrst map each of them to their corresponding CPDAG and measure the SHD between the two.

Original: This metric is useful to compare algorithms which only outputs a CPDAG like GSF, GES and PC to other methods which outputs a DAG.
Revised: This metric is useful to compare algorithms which only outputs a CPDAG like GSF, GES and PC to other methods which outputs a DAG.

Original: Results are reported in Table 3.
Revised: Results are reported in Table 3.

Original: [Table]
Revised: [Table]

Original: First, all methods perform worse than what was reported for graphs of similar size in Section 4.1, both in terms of SID and SHD.
Revised: First, all methods perform worse than what was reported for graphs of similar size in Section 4.1, both in terms of SID and SHD.

Original: This might be due to the lack of identiﬁability guarantees we face in applications.
Revised: This might be due to the lack of identiﬁability guarantees we face in applications.

Original: On the protein data set, GraN-DAG outperforms CAM in terms of SID (which differs from the general trend of Section 4.1) and arrive almost on par in terms of SHD and SHD-C. On this data set, DAG-GNN has a reasonable performance, beating GraN-DAG in SID, but not in SHD.
Revised: On the protein data set, GraN-DAG outperforms CAM in terms of SID (which differs from the general trend of Section 4.1) and arrive almost on par in terms of SHD and SHD-C. On this data set, DAG-GNN has a reasonable performance, beating GraN-DAG in SID, but not in SHD.

Original: On SynTReN, GraN-DAG obtains the best SHD but not the best SID.
Revised: On SynTReN, GraN-DAG obtains the best SHD but not the best SID.

Original: Overall, GraN-DAG is always competitive with the best methods of each task.
Revised: Overall, GraN-DAG is always competitive with the best methods of each task.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Most methods for structure learning from observational data make use of some identiﬁability results similar to the ones raised in Section 2.2.
Revised: Most methods for structure learning from observational data make use of some identiﬁability results similar to the ones raised in Section 2.2.

Original: Roughly speaking, there are two classes of methods: independence-based and score-based methods.
Revised: Roughly speaking, there are two classes of methods: independence-based and score-based methods.

Original: GraN-DAG falls into the second class.
Revised: GraN-DAG falls into the second class.

Original: Score-based methods (Koller & Friedman, 2009; Peters et al., 2017) cast the problem of structure learning as an optimization problem over the space of structures (DAGs or CPDAGs).
Revised: Score-based methods (Koller & Friedman, 2009; Peters et al., 2017) cast the problem of structure learning as an optimization problem over the space of structures (DAGs or CPDAGs).

Original: Many popular algorithms tackle the combinatorial nature of the problem by performing a form of greedy search.
Revised: Many popular algorithms tackle the combinatorial nature of the problem by performing a form of greedy search.

Original: GES (Chickering, 2003) is a popular example.
Revised: GES (Chickering, 2003) is a popular example.

Original: It usually assumes a linear parametric model with
Revised: It usually assumes a linear parametric model with

Original: Gaussian noise and greedily search the space of CPDAGs in order to optimize the Bayesian information criterion.
Revised: Gaussian noise and greedily search the space of CPDAGs in order to optimize the Bayesian information criterion.

Original: GSF (Huang et al., 2018), is based on the same search algorithm as GES, but uses a generalized score function which can model nonlinear relationships.
Revised: GSF (Huang et al., 2018), is based on the same search algorithm as GES, but uses a generalized score function which can model nonlinear relationships.

Original: Other greedy approaches rely on parametric assumptions which render G fully identiﬁable.
Revised: Other greedy approaches rely on parametric assumptions which render G fully identiﬁable.

Original: For example, Peters & B ¨ uhlman (2014) relies on a linear Gaussian model with equal variances to render the DAG identiﬁable.
Revised: For example, Peters & B¨uhlman (2014) relies on a linear Gaussian model with equal variances to render the DAG identiﬁable.

Original: RESIT (Peters et al., 2014), assumes nonlinear relationships with additive Gaussian noise and greedily maximizes an independence-based score.
Revised: RESIT (Peters et al., 2014), assumes nonlinear relationships with additive Gaussian noise and greedily maximizes an independence-based score.

Original: However, RESIT does not scale well to graph of more than 20 nodes.
Revised: However, RESIT does not scale well to graph of more than 20 nodes.

Original: CAM (B¨uhlmann et al., 2014) decouples the search for the optimal node ordering from the parents selection for each node and assumes an additive noise model (ANM) (Peters et al., 2017) in which the nonlinear functions are additive.
Revised: CAM (B¨uhlmann et al., 2014) decouples the search for the optimal node ordering from the parents selection for each node and assumes an additive noise model (ANM) (Peters et al., 2017) in which the nonlinear functions are additive.

Original: As mentioned in Section 2.3, NOTEARS, proposed in Zheng et al.
Revised: As mentioned in Section 2.3, NOTEARS, proposed in Zheng et al.

Original: (2018), tackles the problem of ﬁnding an optimal DAG as a continuous constrained optimization program.
Revised: (2018), tackles the problem of ﬁnding an optimal DAG as a continuous constrained optimization program.

Original: This is a drastic departure from previous combinatorial approaches which enables the application of well studied numerical solvers for continuous optimizations.
Revised: This is a drastic departure from previous combinatorial approaches which enables the application of well studied numerical solvers for continuous optimizations.

Original: Recently, Yu et al.
Revised: Recently, Yu et al.

Original: (2019) proposed DAG-GNN, a graph neural network architecture (GNN) which can be used to learn DAGs via the maximization of an evidence lower bound.
Revised: (2019) proposed DAG-GNN, a graph neural network architecture (GNN) which can be used to learn DAGs via the maximization of an evidence lower bound.

Original: By design, a GNN makes use of parameter sharing which we hypothesize is not well suited for most DAG learning tasks.
Revised: By design, a GNN makes use of parameter sharing which we hypothesize is not well suited for most DAG learning tasks.

Original: To the best of our knowledge, DAG-GNN is the ﬁrst approach extending the NOTEARS algorithm for structure learning to support nonlinear relationships.
Revised: To the best of our knowledge, DAG-GNN is the ﬁrst approach extending the NOTEARS algorithm for structure learning to support nonlinear relationships.

Original: Although Yu et al.
Revised: Although Yu et al.

Original: (2019) provides empirical comparisons to linear approaches, namely NOTEARS and FGS (a faster extension of GES)
Revised: (2019) provides empirical comparisons to linear approaches, namely NOTEARS and FGS (a faster extension of GES)

Original: (Ramsey et al., 2017), comparisons to greedy approaches supporting nonlinear relationships such as CAM and GSF are missing.
Revised: (Ramsey et al., 2017), comparisons to greedy approaches supporting nonlinear relationships such as CAM and GSF are missing.

Original: Moreover, GraN-DAG signiﬁcantly outperforms DAG-GNN on our benchmarks.
Revised: Moreover, GraN-DAG signiﬁcantly outperforms DAG-GNN on our benchmarks.

Original: There exists certain score-based approaches which uses integer linear programming (ILP) (Jaakkola et al., 2010; Cussens, 2011) which internally solve continuous linear relaxations.
Revised: There exists certain score-based approaches which uses integer linear programming (ILP) (Jaakkola et al., 2010; Cussens, 2011) which internally solve continuous linear relaxations.

Original: Connections between such methods and the continuous constrained approaches are yet to be explored.
Revised: Connections between such methods and the continuous constrained approaches are yet to be explored.

Original: When used with the additive Gaussian noise assumption, the theoretical guarantee of GraN-DAG rests on the identiﬁability of nonlinear Gaussian ANMs.
Revised: When used with the additive Gaussian noise assumption, the theoretical guarantee of GraN-DAG rests on the identiﬁability of nonlinear Gaussian ANMs.

Original: Analogously to CAM and NOTEARS, this guarantee holds only if the correct identiﬁability assumptions hold in the data and if the score maximization problem is solved exactly (which is not the case in all three algorithms).
Revised: Analogously to CAM and NOTEARS, this guarantee holds only if the correct identiﬁability assumptions hold in the data and if the score maximization problem is solved exactly (which is not the case in all three algorithms).

Original: DAG-GNN provides no theoretical justiﬁcation for its approach.
Revised: DAG-GNN provides no theoretical justiﬁcation for its approach.

Original: NOTEARS and CAM are designed to handle what is sometimes called the high-dimensional setting in which the number of samples is signiﬁcantly smaller than the number of nodes.
Revised: NOTEARS and CAM are designed to handle what is sometimes called the high-dimensional setting in which the number of samples is signiﬁcantly smaller than the number of nodes.

Original: B¨uhlmann et al.
Revised: B¨uhlmann et al.

Original: (2014) provides consistency results for
Revised: (2014) provides consistency results for

Original: CAM in this setting.
Revised: CAM in this setting.

Original: GraN-DAG and DAG-GNN was not designed with this setting in mind and would most likely fail if confronted to it.
Revised: GraN-DAG and DAG-GNN were not designed with this setting in mind and would most likely fail if confronted to it.

Original: Solutions for ﬁtting a neural network on less data points than input dimensions are not common in the NN literature.
Revised: Solutions for ﬁtting a neural network on less data points than input dimensions are not common in the NN literature.

Original: Methods for causal discovery using NNs have already been proposed.
Revised: Methods for causal discovery using NNs have already been proposed.

Original: SAM (Kalainathan et al., 2018) learns conditional NN generators using adversarial losses but does not enforce acyclicity.
Revised: SAM (Kalainathan et al., 2018) learns conditional NN generators using adversarial losses but does not enforce acyclicity.

Original: CGNN (Goudet et al., 2018), when used for multivariate data, requires an initial skeleton to learn the different functional relationships.
Revised: CGNN (Goudet et al., 2018), when used for multivariate data, requires an initial skeleton to learn the different functional relationships.

Original: GraN-DAG has strong connections with MADE (Germain et al., 2015), a method used to learn distributions using a masked NN which enforces the so-called autoregressive property .
Revised: GraN-DAG has strong connections with MADE (Germain et al., 2015), a method used to learn distributions using a masked NN which enforces the so-called autoregressive property .

Original: The autoregressive property and acyclicity are in fact equivalent.
Revised: The autoregressive property and acyclicity are in fact equivalent.

Original: MADE does not learn the weight masking, it ﬁxes it at the beginning of the procedure.
Revised: MADE does not learn the weight masking, it ﬁxes it at the beginning of the procedure.

Original: GraN-DAG could be used with a unique NN taking as input all variables and outputting parameters for all conditional distributions.
Revised: GraN-DAG could be used with a unique NN taking as input all variables and outputting parameters for all conditional distributions.

Original: In this case, it would be similar to MADE, except the variable ordering would be learned from data instead of ﬁxed a priori .
Revised: In this case, it would be similar to MADE, except the variable ordering would be learned from data instead of ﬁxed a priori .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The continuous constrained approach to structure learning has the advantage of being more global than other approximate greedy methods (since it updates all edges at each step based on the gradient of the score but also the acyclicity constraint) and allows to replace task-speciﬁc greedy algorithms by appropriate off-the-shelf numerical solvers.
Revised: The continuous constrained approach to structure learning has the advantage of being more global than other approximate greedy methods (since it updates all edges at each step based on the gradient of the score but also the acyclicity constraint) and allows to replace task-speciﬁc greedy algorithms by appropriate off-the-shelf numerical solvers.

Original: In this work, we have introduced GraN-DAG, a novel score-based approach for structure learning supporting nonlinear relationships while leveraging a continuous optimization paradigm.
Revised: In this work, we have introduced GraN-DAG, a novel score-based approach for structure learning supporting nonlinear relationships while leveraging a continuous optimization paradigm.

Original: The method rests on a novel characterization of acyclicity for NNs based on the work of Zheng et al.
Revised: The method rests on a novel characterization of acyclicity for NNs based on the work of Zheng et al.

Original: We showed GraN-DAG outperforms other gradient-based approaches, namely NOTEARS and its recent nonlinear extension DAG-GNN, on the synthetic data sets considered in Section 4.1 while being competitive on real and pseudo-real data sets of Section 4.2.
Revised: We showed GraN-DAG outperforms other gradient-based approaches, namely NOTEARS and its recent nonlinear extension DAG-GNN, on the synthetic data sets considered in Section 4.1 while being competitive on real and pseudo-real data sets of Section 4.2.

Original: Compared to greedy approaches, GraN-DAG is competitive across all datasets considered.
Revised: Compared to greedy approaches, GraN-DAG is competitive across all datasets considered.

Original: To the best of our knowledge, GraN-DAG is the ﬁrst approach leveraging the continuous paradigm introduced in Zheng et al.
Revised: To the best of our knowledge, GraN-DAG is the ﬁrst approach leveraging the continuous paradigm introduced in Zheng et al.

Original: (2018) which has been shown to be competitive with state of the art methods supporting nonlinear relationships
Revised: (2018) which has been shown to be competitive with state of the art methods supporting nonlinear relationships.

Original: A A PPENDIX A. O PTIMIZATION
Revised: A CKNOWLEDGMENTS

Revised: This research was partially supported by the Canada CIFAR AI Chair Program and by a Google Focused Research award. The authors would like to thank R´emi Le Priol, Tatjana Chavdarova,

Revised: Charles Guille-Escuret, Nicolas Gagn´e and Yoshua Bengio for insightful discussions as well as

Revised: Alexandre Drouin and Florian Bordes for technical support.

Revised: The experiments were in part enabled by computational resources provided by Calcul Qu´ebec, Compute Canada and Element AI.

Revised: A A PPENDIX A. O PTIMIZATION

Original: Let us recall the augmented Lagrangian:
Revised: Let us recall the augmented Lagrangian:

Original: [Equation]
Revised: [Equation]

Original: where λ t and µ t are the Lagrangian and penalty coefﬁcients of the t th subproblem, respectively.
Revised: where λ t and µ t are the Lagrangian and penalty coefﬁcients of the t th subproblem, respectively.

Original: In all our experiments, we initialize those coefﬁcients using λ 0 = 0 and µ 0 = 10 − 3 .
Revised: In all our experiments, we initialize those coefﬁcients using λ 0 = 0 and µ 0 = 10 − 3 .

Original: We approximately solve each non-convex subproblem using RMSprop (Tieleman & Hinton, 2012), a stochastic gradient descent variant popular for NNs.
Revised: We approximately solve each non-convex subproblem using RMSprop (Tieleman & Hinton, 2012), a stochastic gradient descent variant popular for NNs.

Original: We use the following gradient estimate:
Revised: We use the following gradient estimate:

Original: [Equation]
Revised: [Equation]

Original: with ˆ L B ( φ, λ t , µ t ) (cid:44) 1 | B | (cid:88)
Revised: with ˆ L B ( φ, λ t , µ t ) (cid:44) 1 | B | (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where B is a minibatch sampled from the data set and | B | is the minibatch size.
Revised: where B is a minibatch sampled from the data set and | B | is the minibatch size.

Original: The gradient estimate ∇ φ ˆ L B ( φ, λ t , µ t ) can be computed using standard deep learning libraries.
Revised: The gradient estimate ∇ φ ˆ L B ( φ, λ t , µ t ) can be computed using standard deep learning libraries.

Original: We consider a subproblem has converged when ˆ L H ( φ, λ t , µ t ) evaluated on a held-out data set H stops increasing.
Revised: We consider a subproblem has converged when ˆ L H ( φ, λ t , µ t ) evaluated on a held-out data set H stops increasing.

Original: Let φ ∗ t be the approximate solution to subproblem t .
Revised: Let φ ∗ t be the approximate solution to subproblem t .

Original: Then, λ t and µ t are updated according to the following rule:
Revised: Then, λ t and µ t are updated according to the following rule:

Original: [Equation]
Revised: [Equation]

Original: otherwise (14) with η = 10 and γ = 0 .
Revised: otherwise (14) with η = 10 and γ = 0 .

Original: Each subproblem t is initialized using the previous subproblem solution φ ∗ t − 1 .
Revised: Each subproblem t is initialized using the previous subproblem solution φ ∗ t − 1 .

Original: The augmented Lagrangian method stops when h ( φ ) ≤ 10 − 8 .
Revised: The augmented Lagrangian method stops when h ( φ ) ≤ 10 − 8 .

Original: Total number of iterations before augmented Lagrangian converges:
Revised: Total number of iterations before augmented Lagrangian converges:

Original: In GraN-DAG and
Revised: In GraN-DAG and

Original: NOTEARS, every subproblem is approximately solved using an iterative algorithm.
Revised: NOTEARS, every subproblem is approximately solved using an iterative algorithm.

Original: Let T be the number of subproblems solved before the convergence of the augmented Lagrangian.
Revised: Let T be the number of subproblems solved before the convergence of the augmented Lagrangian.

Original: For a given subproblem t , let K t be the number of iterations executed to approximately solve it.
Revised: For a given subproblem t , let K t be the number of iterations executed to approximately solve it.

Original: Let
Revised: Let

Original: I = (cid:80) Tt =1 K t be the total number of iterations before the augmented Lagrangian converges.
Revised: I = (cid:80) Tt =1 K t be the total number of iterations before the augmented Lagrangian converges.

Original: Table 4 reports the total number of iterations I for GraN-DAG and NOTEARS, averaged over ten data sets.
Revised: Table 4 reports the total number of iterations I for GraN-DAG and NOTEARS, averaged over ten data sets.

Original: Note that the matrix exponential is evaluated once per iteration.
Revised: Note that the matrix exponential is evaluated once per iteration.

Original: Even though GraN-DAG uses a stochastic gradient algorithm, it requires less iterations than NOTEARS which uses a batch proximal quasi-Newton method.
Revised: Even though GraN-DAG uses a stochastic gradient algorithm, it requires less iterations than NOTEARS which uses a batch proximal quasi-Newton method.

Original: We hypothesize early stopping avoids having to wait until full convergence before moving to the next subproblem, hence reducing the total number of iterations.
Revised: We hypothesize early stopping avoids having to wait until full convergence before moving to the next subproblem, hence reducing the total number of iterations.

Original: Note that GraN-DAG total run time is still larger than NOTEARS due to its gradient requiring more computation to evaluate (total runtime ≈ 10 minutes against ≈ 1 minute for 20 nodes graphs and ≈ 4 hours against ≈ 1 hour for 100 nodes graphs).
Revised: Note that GraN-DAG total run time is still larger than NOTEARS due to its gradient requiring more computation to evaluate (total runtime ≈ 10 minutes against ≈ 1 minute for 20 nodes graphs and ≈ 4 hours against ≈ 1 hour for 100 nodes graphs).

Original: GraN-DAG runtime on 100 nodes graphs can be roughly halfed when executed on GPU.[Table] A.
Revised: GraN-DAG runtime on 100 nodes graphs can be roughly halved when executed on GPU.

Original: [Table]
Revised: [Table]

Original:  T HRESHOLDING TO ENSURE ACYCLICITY
Revised: A. T HRESHOLDING TO ENSURE ACYCLICITY

Original: The augmented Lagrangian outputs φ ∗ T where T is the number of subproblems solved before declaring convergence.
Revised: The augmented Lagrangian outputs φ ∗ T where T is the number of subproblems solved before declaring convergence.

Original: Note that the weighted adjacency matrix A φ ∗ T will most likely not represent an acyclic graph, even if we threshold as we learn, as explained in Section 3.4.
Revised: Note that the weighted adjacency matrix A φ ∗ T will most likely not represent an acyclic graph, even if we threshold as we learn, as explained in Section 3.4.

Original: We need to remove additional edges to obtain a DAG (edges are removed using the mask presented in Section 3.4).
Revised: We need to remove additional edges to obtain a DAG (edges are removed using the mask presented in Section 3.4).

Original: One option would be to remove edges one by one until a DAG is obtained, starting from the edge ( i, j ) with the lowest ( A φ ∗ T ) ij up to the edge with the highest ( A φ ∗ T ) ij .
Revised: One option would be to remove edges one by one until a DAG is obtained, starting from the edge ( i, j ) with the lowest ( A φ ∗ T ) ij up to the edge with the highest ( A φ ∗ T ) ij .

Original: This amounts to gradually increasing the threshold (cid:15) until A φ ∗ T is acyclic.
Revised: This amounts to gradually increasing the threshold (cid:15) until A φ ∗ T is acyclic.

Original: However, this approach has the following ﬂaw: It is possible to have ( A φ ∗ T ) ij signiﬁcantly higher than zero while having θ ( j ) almost completely independent of variable X i .
Revised: However, this approach has the following ﬂaw: It is possible to have ( A φ ∗ T ) ij signiﬁcantly higher than zero while having θ ( j ) almost completely independent of variable X i .

Original: This can happen for at least two reasons.
Revised: This can happen for at least two reasons.

Original: First, the NN paths from input i to output k might end up cancelling each others, rendering the input i inactive.
Revised: First, the NN paths from input i to output k might end up cancelling each others, rendering the input i inactive.

Original: Second, some neurons of the
Revised: Second, some neurons of the

Original: NNs might always be saturated for the observed range of inputs, rendering some NN paths effectively inactive without being inactive in the sense described in Section 3.1.
Revised: NNs might always be saturated for the observed range of inputs, rendering some NN paths effectively inactive without being inactive in the sense described in Section 3.1.

Original: Those two observations illustrate the fact that having ( A φ ∗ T ) ij = 0 is only a sufﬁcient condition to have θ ( j ) independent of variable X i and not a necessary one.
Revised: Those two observations illustrate the fact that having ( A φ ∗ T ) ij = 0 is only a sufﬁcient condition to have θ ( j ) independent of variable X i and not a necessary one.

Original: To avoid this issue, we consider the following alternative.
Revised: To avoid this issue, we consider the following alternative.

Original: Consider the function L : R d (cid:55)→ R d which maps all d variables to their respective conditional likelihoods, i.e. L i ( X ) (cid:44) p
Revised: Consider the function L : R d (cid:55)→ R d which maps all d variables to their respective conditional likelihoods, i.e. L i ( X ) (cid:44) p

Original: i ( X i |
Revised: i ( X i |

Original: X π φ ∗ T i ) ∀ i .
Revised: X π φ ∗ T i ) ∀ i .

Original: We consider the following expected Jacobian matrix
Revised: We consider the following expected Jacobian matrix

Original: [Equation]
Revised: [Equation]

Original:  (15) where (cid:12)(cid:12) ∂
Revised: (cid:62) (15) where (cid:12)(cid:12) ∂

Original: L ∂X (cid:12)(cid:12) is the Jacobian matrix of L evaluated at X , in absolute value (element-wise).
Revised: L ∂X (cid:12)(cid:12) is the Jacobian matrix of L evaluated at X , in absolute value (element-wise).

Original: Similarly to ( A φ ∗ T ) ij , the entry J ij can be loosely interpreted as the strength of edge ( i, j ) .
Revised: Similarly to ( A φ ∗ T ) ij , the entry J ij can be loosely interpreted as the strength of edge ( i, j ) .

Original: We propose removing edges starting from the lowest J ij to the highest, stopping as soon as acyclicity is achieved.
Revised: We propose removing edges starting from the lowest J ij to the highest, stopping as soon as acyclicity is achieved.

Original: We believe J is better than A φ ∗ T at capturing which NN inputs are effectively inactive since it takes into account NN paths cancelling each others and saturated neurons.
Revised: We believe J is better than A φ ∗ T at capturing which NN inputs are effectively inactive since it takes into account NN paths cancelling each others and saturated neurons.

Original: Empirically, we found that using J instead of A φ ∗ T yields better results, and thus we report the results with J in this paper.
Revised: Empirically, we found that using J instead of A φ ∗ T yields better results, and thus we report the results with J in this paper.

Original: A. P RELIMINARY NEIGHBORHOOD SELECTION AND DAG P RUNING
Revised: A. P RELIMINARY NEIGHBORHOOD SELECTION AND DAG P RUNING

Original: PNS:
Revised: PNS:

Original: For graphs of 50 nodes or more, GraN-DAG performs a preliminary neighborhood selection (PNS) similar to what has been proposed in B ¨ uhlmann et al.
Revised: For graphs of 50 nodes or more, GraN-DAG performs a preliminary neighborhood selection (PNS) similar to what has been proposed in B ¨ uhlmann et al.

Original: This procedure applies a variable selection method to get a set of possible parents for each node.
Revised: This procedure applies a variable selection method to get a set of possible parents for each node.

Original: This is done by ﬁtting an extremely randomized trees (Geurts et al., 2006) (using ExtraTreesRegressor from scikit-learn ) for each variable against all the other variables.
Revised: This is done by ﬁtting an extremely randomized trees (Geurts et al., 2006) (using ExtraTreesRegressor from scikit-learn ) for each variable against all the other variables.

Original: For each node a feature importance score based on the gain of purity is calculated.
Revised: For each node a feature importance score based on the gain of purity is calculated.

Original: Only nodes that have a feature importance score higher than 0 .
Revised: Only nodes that have a feature importance score higher than 0 .

Original: 75 · mean are kept as potential parent, where mean is the mean of the feature importance scores of all nodes.
Revised: 75 · mean are kept as potential parent, where mean is the mean of the feature importance scores of all nodes.

Original: Although the use of PNS in CAM was motivated by gains in computation time, GraN-DAG uses it to avoid overﬁtting, without reducing the computation time.
Revised: Although the use of PNS in CAM was motivated by gains in computation time, GraN-DAG uses it to avoid overﬁtting, without reducing the computation time.

Original: Pruning: Once the thresholding is performed and a DAG is obtained as described in A.2, GraNDAG performs a pruning step identical to CAM (B¨uhlmann et al., 2014) in order to remove spurious edges.
Revised: Pruning: Once the thresholding is performed and a DAG is obtained as described in A.2, GraNDAG performs a pruning step identical to CAM (B¨uhlmann et al., 2014) in order to remove spurious edges.

Original: We use the implementation of B¨uhlmann et al.
Revised: We use the implementation of B¨uhlmann et al.

Original: based on the R function gamboost from the mboost package.
Revised: based on the R function gamboost from the mboost package.

Original: For each variable X i , a generalized additive model is ﬁtted against the current parents of X i and a signiﬁcance test of covariance is applied.
Revised: For each variable X i , a generalized additive model is ﬁtted against the current parents of X i and a signiﬁcance test of covariates is applied.

Original: Parents with a p-value higher than 0.001 are removed from the parent set.
Revised: Parents with a p-value higher than 0.001 are removed from the parent set.

Original: Similarly to what B¨uhlmann et al.
Revised: Similarly to what B¨uhlmann et al.

Original: (2014) observed, this pruning phase generally has the effect of greatly reducing the SHD without considerably changing the SID.
Revised: (2014) observed, this pruning phase generally has the effect of greatly reducing the SHD without considerably changing the SID.

Original: Ablation study:
Revised: Ablation study:

Original: In Table 5, we present an ablation study which shows the effect of adding PNS and pruning to GraN-DAG on different performance metrics and on the negative log-likelihood (NLL) of the training and validation set.
Revised: In Table 5, we present an ablation study which shows the effect of adding PNS and pruning to GraN-DAG on different performance metrics and on the negative log-likelihood (NLL) of the training and validation set.

Original: Note that, before computing both NLL, we reset all parameters of GraN-DAG except the mask and retrained the model on the training set without any acyclicity constraint (acyclicity is already ensure by the masks at this point).
Revised: Note that, before computing both NLL, we reset all parameters of GraN-DAG except the mask and retrained the model on the training set without any acyclicity constraint (acyclicity is already ensure by the masks at this point).

Original: This retraining procedure is important since the pruning removes edges (i.e. some additional NN inputs are masked) and it affects the likelihood of the model (hence the need to retrain).
Revised: This retraining procedure is important since the pruning removes edges (i.e. some additional NN inputs are masked) and it affects the likelihood of the model (hence the need to retrain).

Original: [Table]
Revised: [Table]

Original: A ﬁrst observation is that adding PNS and pruning improve the NLL on the validation set while deteriorating the NLL on the training set, showing that those two steps are indeed reducing overﬁtting.
Revised: A ﬁrst observation is that adding PNS and pruning improve the NLL on the validation set while deteriorating the NLL on the training set, showing that those two steps are indeed reducing overﬁtting.

Original: Secondly, the effect on SHD is really important while the effect on SID is almost nonexistent.
Revised: Secondly, the effect on SHD is really important while the effect on SID is almost nonexistent.

Original: This can be explained by the fact that SID has more to do with the ordering of the nodes than with false positive edges.
Revised: This can be explained by the fact that SID has more to do with the ordering of the nodes than with false positive edges.

Original: For instance, if we have a complete DAG with a node ordering coherent with the ground truth graph, the SID is zero, but the SHD is not due to all the false positive edges.
Revised: For instance, if we have a complete DAG with a node ordering coherent with the ground truth graph, the SID is zero, but the SHD is not due to all the false positive edges.

Original: Without the regularizing effect of PNS and pruning, GraN-DAG manages to ﬁnd a DAG with a good ordering but with many spurious edges (explaining the poor SHD, the good SID and the big gap between the
Revised: Without the regularizing effect of PNS and pruning, GraN-DAG manages to ﬁnd a DAG with a good ordering but with many spurious edges (explaining the poor SHD, the good SID and the big gap between the

Original: NLL of the training set and validation set).
Revised: NLL of the training set and validation set).

Original: PNS and pruning helps reducing the number of spurious edges, hence improving SHD.
Revised: PNS and pruning helps reducing the number of spurious edges, hence improving SHD.

Original: We also implemented PNS and pruning for NOTEARS and DAG-GNN to see whether their performance could also be improved.
Revised: We also implemented PNS and pruning for NOTEARS and DAG-GNN to see whether their performance could also be improved.

Original: Table 6 reports an ablation study for DAG-GNN and NOTEARS.
Revised: Table 6 reports an ablation study for DAG-GNN and NOTEARS.

Original: First, the SHD improvement is not as important as for GraN-DAG and is almost not statistically signiﬁcant.
Revised: First, the SHD improvement is not as important as for GraN-DAG and is almost not statistically signiﬁcant.

Original: The improved SHD does not come close to performance of GraN-DAG.
Revised: The improved SHD does not come close to performance of GraN-DAG.

Original: Second, PNS and pruning do not have a signiﬁcant effect of SID, as was the case for GraN-DAG.
Revised: Second, PNS and pruning do not have a signiﬁcant effect of SID, as was the case for GraN-DAG.

Original: The lack of improvement for those methods is probably due to the fact that they are not overﬁtting like GraNDAG, as the training and validation (unregularized) scores shows.
Revised: The lack of improvement for those methods is probably due to the fact that they are not overﬁtting like GraNDAG, as the training and validation (unregularized) scores shows.

Original: NOTEARS captures only linear relationships, thus it will have a hard time overﬁtting nonlinear data and DAG-GNN uses a strong form of parameter sharing between its conditional densities which possibly cause underﬁtting in a setup where all the parameters of the conditionals are sampled independently.
Revised: NOTEARS captures only linear relationships, thus it will have a hard time overﬁtting nonlinear data and DAG-GNN uses a strong form of parameter sharing between its conditional densities which possibly cause underﬁtting in a setup where all the parameters of the conditionals are sampled independently.

Original: Moreover, DAG-GNN and NOTEARS threshold aggressively their respective weighted adjacency matrix at the end of training (with the default parameters used in the code), which also acts as a form of heavy regularization, and allow them to remove many spurious edges.
Revised: Moreover, DAG-GNN and NOTEARS threshold aggressively their respective weighted adjacency matrix at the end of training (with the default parameters used in the code), which also acts as a form of heavy regularization, and allow them to remove many spurious edges.

Original: GraN-DAG without
Revised: GraN-DAG without

Original: PNS and pruning does not threshold as strongly by default which explains the high SHD of Table 5.
Revised: PNS and pruning does not threshold as strongly by default which explains the high SHD of Table 5.

Original: To test this explanation, we removed all edges ( i , j ) for which ( A φ ) ij < 0 .
Revised: To test this explanation, we removed all edges ( i , j ) for which ( A φ ) ij < 0 .

Original: 3 4 for GraN-DAG and obtained an SHD of 29.4 ± 15.9 and an SID of 85.6 ± 45.7, showing a signiﬁcant improvement over
Revised: 3 3 for GraN-DAG and obtained an SHD of 29.4 ± 15.9 and an SID of 85.6 ± 45.7, showing a signiﬁcant improvement over

Original: NOTEARS and DAG-GNN, even without PNS and pruning.
Revised: NOTEARS and DAG-GNN, even without PNS and pruning.

Original: [Table]
Revised: [Table]

Original: A. L ARGE S AMPLE S IZE E XPERIMENT
Revised: A. L ARGE S AMPLE S IZE E XPERIMENT

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we test the bias-variance hypothesis which attempts to explain why CAM is on par with GraN-DAG on Gauss-ANM data even if its model wrongly assumes that the f j functions are additive.
Revised: we test the bias-variance hypothesis which attempts to explain why CAM is on par with GraN-DAG on Gauss-ANM data even if its model wrongly assumes that the f j functions are additive.

Original: Table 7 reports the performance of GraN-DAG and CAM for different sample sizes.
Revised: Table 7 reports the performance of GraN-DAG and CAM for different sample sizes.

Original: We can see that, as the sample size grows, GraN-DAG ends up outperforming CAM in terms of
Revised: We can see that, as the sample size grows, GraN-DAG ends up outperforming CAM in terms of

Original: SID while staying on par in terms of SHD.
Revised: SID while staying on par in terms of SHD.

Original: We explain this observation by the fact that a larger sample size reduces variance for GraN-DAG thus allowing it to leverage its greater capacity against
Revised: We explain this observation by the fact that a larger sample size reduces variance for GraN-DAG thus allowing it to leverage its greater capacity against

Original: CAM which is stuck with its modelling bias.
Revised: CAM which is stuck with its modelling bias.

Original: Both algorithms were run with their respective default hyperparameter combination.
Revised: Both algorithms were run with their respective default hyperparameter combination.

Original: This experiment suggests GraN-DAG could be an appealing option in settings where the sample size is substantial.
Revised: This experiment suggests GraN-DAG could be an appealing option in settings where the sample size is substantial.

Original: The present paper focuses on sample sizes typically encountered in the structure/causal learning litterature and leave this question for future work
Revised: The present paper focuses on sample sizes typically encountered in the structure/causal learning litterature and leave this question for future work

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  D ETAILS ON DATA SETS GENERATION
Revised: A. D ETAILS ON DATA SETS GENERATION

Original: Synthetic data sets: For each data set type, 10 data sets are sampled with 1000 examples each.
Revised: Synthetic data sets: For each data set type, 10 data sets are sampled with 1000 examples each.

Original: As the synthetic data introduced in Section 4.1, for each data set, a ground truth DAG G is randomly sampled following the 
Revised: As the synthetic data introduced in Section 4.1, for each data set, a ground truth DAG G is randomly sampled following the ER

Original: ER scheme and then the data is generated.
Revised: scheme and then the data is generated.

Original: Unless otherwise stated, all root variables are sampled from U [ − 1 , 1] .
Revised: Unless otherwise stated, all root variables are sampled from U [ − 1 , 1] .

Original: •  LIN is generated following X j |
Revised: • Gauss-ANM is generated following X j := f j ( X π G j ) + N j ∀ j with mutually independent noises N j ∼ N (0 , σ 2 j ) ∀ j where the functions f j are independently sampled from a Gaussian process with a unit bandwidth RBF kernel and σ 2 j ∼ U [0 . 4 , 0 . 8] . Source nodes are Gaussian with zero mean and variance sampled from U [1 , 2] • LIN is generated following X j |

Original: X π G j ∼ w Tj X
Revised: X π G j ∼ w T

Original:  π G j + 0 . 2 · N (0 , σ 2 j ) ∀ j where σ 2 j ∼ U [1 , 2] and w j is a vector of | π G j | coefﬁcients each sampled from U [0 , 1] .
Revised: j X π G j + 0 . 2 · N (0 , σ 2 j ) ∀ j where σ 2 j ∼ U [1 , 2] and w j is a vector of | π G j | coefﬁcients each sampled from U [0 , 1] .

Original: • ADD-FUNC is generated following X j | X π G j ∼ (cid:80) i ∈
Revised: • ADD-FUNC is generated following X j | X π G j ∼ (cid:80) i ∈

Original: π G j f j,i ( X i )
Revised: π G j f j,i ( X i )

Original: + 0 .
Revised: + 0 .

Original: 2 · N (0 , σ 2 j ) ∀ j where σ 2 j ∼ U [1 , 2] and the functions f j,i are independently sampled from a Gaussian process with bandwidth one.
Revised: 2 · N (0 , σ 2 j ) ∀ j where σ 2 j ∼ U [1 , 2] and the functions f j,i are independently sampled from a Gaussian process with bandwidth one.

Original: This model is adapted from B ¨ uhlmann et al.
Revised: This model is adapted from B¨uhlmann et al.

Original: • PNL-GP is generated following X j | X π G j ∼ σ ( f j ( X π G j ) + Laplace (0 , l j ))
Revised: • PNL-GP is generated following X j | X π G j ∼ σ ( f j ( X π G j ) + Laplace (0 , l j ))

Original: ∀ j with the functions
Revised: ∀ j with the functions

Original: f j independently sampled from a Gaussian process with bandwidth one and l j ∼ U [0 , 1] .
Revised: f j independently sampled from a Gaussian process with bandwidth one and l j ∼ U [0 , 1] .

Original: In the two-variable case, this model is identiﬁable following the Corollary 9 from Zhang & Hyv¨arinen (2009).
Revised: In the two-variable case, this model is identiﬁable following the Corollary 9 from Zhang & Hyv¨arinen (2009).

Original: To get identiﬁability according to this corollary, it is important to use non-Gaussian noise, explaining our design choices.
Revised: To get identiﬁability according to this corollary, it is important to use non-Gaussian noise, explaining our design choices.

Original: • PNL-MULT is generated following X j | X π G j ∼ exp(log( (cid:80) i ∈
Revised: • PNL-MULT is generated following X j | X π G j ∼ exp(log( (cid:80) i ∈

Original: π G j X i )
Revised: π G j X i )

Original: + |N (0 , σ 2 j ) | ) ∀
Revised: + |N (0 , σ 2 j ) | ) ∀

Original: j where σ 2 j ∼ U [0 , 1] .
Revised: j where σ 2 j ∼ U [0 , 1] .

Original: Root variables are sampled from U [0 , 2] .
Revised: Root variables are sampled from U [0 , 2] .

Original: This model is adapted from Zhang et al.
Revised: This model is adapted from Zhang et al.

Original: SynTReN: Ten datasets have been generated using the SynTReN generator ( http: //bioinformatics.intec.ugent.be/kmarchal/SynTReN/index.html ) using the software default parameters except for the probability for complex 2-regulator interactions that was set to 1 and the random seeds used were 0 to 9.
Revised: SynTReN: Ten datasets have been generated using the SynTReN generator ( http: //bioinformatics.intec.ugent.be/kmarchal/SynTReN/index.html ) using the software default parameters except for the probability for complex 2-regulator interactions that was set to 1 and the random seeds used were 0 to 9.

Original: Each dataset contains 500 samples and comes from a 20 nodes graph.
Revised: Each dataset contains 500 samples and comes from a 20 nodes graph.

Original: Graph types: Erd˝os - R´enyi (ER) graphs are generated by randomly sampling a topological order and by adding directed edges were it is allowed independently with probability p =
Revised: Graph types: Erd˝os - R´enyi (ER) graphs are generated by randomly sampling a topological order and by adding directed edges were it is allowed independently with probability p =

Original: e d 2 − d were e is the expected number of edges in the resulting DAG.
Revised: e d 2 − d were e is the expected number of edges in the resulting DAG.

Original: Scale-free (SF) graphs were generated using the
Revised: Scale-free (SF) graphs were generated using the

Original: BarabsiAlbert model (Barab´asi & Albert, 1999) which is based on preferential attachment.
Revised: Barab´asi-Albert model (Barab´asi & Albert, 1999) which is based on preferential attachment.

Original: Nodes are added one by one.
Revised: Nodes are added one by one.

Original: Between the new node and the existing nodes, m edges (where m is equal to d or 4 d ) will be added.
Revised: Between the new node and the existing nodes, m edges (where m is equal to d or 4 d ) will be added.

Original: An existing node i have the probability p ( k i ) =
Revised: An existing node i have the probability p ( k i ) =

Original: k i (cid:80) j k j to be chosen, where k i represents the degree of the node i .
Revised: k i (cid:80) j k j to be chosen, where k i represents the degree of the node i .

Original: While ER graphs have a degree distribution following a Poisson distribution, SF graphs have a degree distribution following a power law: few nodes, often called hubs , have a high degree.
Revised: While ER graphs have a degree distribution following a Poisson distribution, SF graphs have a degree distribution following a power law: few nodes, often called hubs , have a high degree.

Original: Barab´asi (2009) have stated that these types of graphs have similar properties to real-world networks which can be found in many different ﬁelds, although these claims remain controversial (Clauset et al., 2009).
Revised: Barab´asi (2009) have stated that these types of graphs have similar properties to real-world networks which can be found in many different ﬁelds, although these claims remain controversial (Clauset et al., 2009).

Original: A. S UPPLEMENTARY EXPERIMENTS
Revised: A. S UPPLEMENTARY EXPERIMENTS

Original: Gauss-ANM: The results for 20 and 100 nodes are presented in Table 8 and 9 using the same GaussANM data set types introduced in Section 4.1.
Revised: Gauss-ANM: The results for 20 and 100 nodes are presented in Table 8 and 9 using the same GaussANM data set types introduced in Section 4.1.

Original: The conclusions drawn remains similar to the 10 and 50 nodes experiments.
Revised: The conclusions drawn remains similar to the 10 and 50 nodes experiments.

Original: For GES and PC, the SHD and SID are respectively presented in Tableand 11.
Revised: For GES and PC, the SHD and SID are respectively presented in Tableand 11.

Original: Their performances do not compare favorably to the GraN-DAG nor CAM.
Revised: Their performances do not compare favorably to the GraN-DAG nor CAM.

Original: Figure 1 shows the entries of the weighted adjacency matrix A φ as training proceeds in a typical run for 10 nodes.
Revised: Figure 1 shows the entries of the weighted adjacency matrix A φ as training proceeds in a typical run for 10 nodes.

Original: LIN & ADD-FUNC: Experiments with LIN and ADD-FUNC data is reported in Table 12 & 13.
Revised: LIN & ADD-FUNC: Experiments with LIN and ADD-FUNC data is reported in Table 12 & 13.

Original: The details of their generation are given in Appendix A.5.
Revised: The details of their generation are given in Appendix A.5.

Original: PNL-GP & PNL-MULT:
Revised: PNL-GP & PNL-MULT:

Original: Table 14 contains the performance of GraN-DAG and other baselines on post nonlinear data discussed in Section 4.1.
Revised: Table 14 contains the performance of GraN-DAG and other baselines on post nonlinear data discussed in Section 4.1.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table]
Revised: [Table] Tables 9 and 14.

Original: [Table]
Revised: search could not ﬁnish within 12 hours,

Original: [Table]
Revised: [Table]

Original:  See 
Revised: data. See Appendix A.7 for details on how to compute SID for CPDAGs

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: We can see that GraN-DAG successfully recovers most edges correctly while keeping few spurious edges.

Original: A. M ETRICS
Revised: A. M ETRICS

Original: SHD takes two partially directed acyclic graphs (PDAG) and counts the number of edge for which the edge type differs in both PDAGs.
Revised: SHD takes two partially directed acyclic graphs (PDAG) and counts the number of edge for which the edge type differs in both PDAGs.

Original: There are four edge types: i ← j , i → j , i −− j and i j .
Revised: There are four edge types: i ← j , i → j , i −− j and i j .

Original: Since this distance is deﬁned over the space of PDAGs, we can use it to compare DAGs with DAGs,
Revised: Since this distance is deﬁned over the space of PDAGs, we can use it to compare DAGs with DAGs,

Original: DAGs with CPDAGs and CPDAGs with CPDAGs.
Revised: DAGs with CPDAGs and CPDAGs with CPDAGs.

Original: When comparing a DAG with a CPDAG, having i ← j instead of i −− j counts as a mistake.
Revised: When comparing a DAG with a CPDAG, having i ← j instead of i −− j counts as a mistake.

Original: SHD-C is very similar to SHD.
Revised: SHD-C is very similar to SHD.

Original: The only difference is that both DAGs are ﬁrst mapped to their respective CPDAGs before measuring the SHD.
Revised: The only difference is that both DAGs are ﬁrst mapped to their respective CPDAGs before measuring the SHD.

Original: Introduced by Peters & B¨uhlmann (2015), SID counts the number of interventional distribution of the form p ( x i | do ( x j = ˆ x j )) that would be miscalculated using the parent adjustment formula (Pearl, 2009) if we were to use the predicted DAG instead of the ground truth DAG to form the parent adjustment set.
Revised: Introduced by Peters & B¨uhlmann (2015), SID counts the number of interventional distribution of the form p ( x i | do ( x j = ˆ x j )) that would be miscalculated using the parent adjustment formula (Pearl, 2009) if we were to use the predicted DAG instead of the ground truth DAG to form the parent adjustment set.

Original: Some care needs to be taken to evaluate the SID for methods outputting a
Revised: Some care needs to be taken to evaluate the SID for methods outputting a

Original: CPDAG such as GES and PC.
Revised: CPDAG such as GES and PC.

Original: Peters & B¨uhlmann (2015) proposes to report the SID of the DAGs which have approximately the minimal and the maximal SID in the Markov equivalence class given by the CPDAG.
Revised: Peters & B¨uhlmann (2015) proposes to report the SID of the DAGs which have approximately the minimal and the maximal SID in the Markov equivalence class given by the CPDAG.

Original: See Peters & B¨uhlmann (2015) for more details.
Revised: See Peters & B¨uhlmann (2015) for more details.

Original: A. H YPERPARAMETERS
Revised: A. H YPERPARAMETERS

Original: All GraN-DAG runs up to this point were performed using the following set of hyperparameters.
Revised: All GraN-DAG runs up to this point were performed using the following set of hyperparameters.

Original: We used RMSprop as optimizer with learning rate of 10 − 2 for the ﬁrst subproblem and 10 − 4 for all subsequent suproblems.
Revised: We used RMSprop as optimizer with learning rate of 10 − 2 for the ﬁrst subproblem and 10 − 4 for all subsequent suproblems.

Original: Each NN has two hidden layers with 10 units (except for the real and pseudo-real data experiments of Table 3 which uses only 1 hidden layer).
Revised: Each NN has two hidden layers with 10 units (except for the real and pseudo-real data experiments of Table 3 which uses only 1 hidden layer).

Original: Leaky-ReLU is used as activation functions.
Revised: Leaky-ReLU is used as activation functions.

Original: The NN are initialized using the initialization scheme proposed in Glorot &
Revised: The NN are initialized using the initialization scheme proposed in Glorot &

Original: Bengio (2010) also known as Xavier initialization .
Revised: Bengio (2010) also known as Xavier initialization .

Original: We used minibatches of 64 samples.
Revised: We used minibatches of 64 samples.

Original: This hyperparameter combination have been selected via a small scale experiment in which many hyperparameter combinations have been tried manually on a single data set of type ER1 with 10 nodes until one yielding a satisfactory SHD was obtained.
Revised: This hyperparameter combination have been selected via a small scale experiment in which many hyperparameter combinations have been tried manually on a single data set of type ER1 with 10 nodes until one yielding a satisfactory SHD was obtained.

Original: Of course in practice one cannot select hyperparameters in this way since we do not have access to the ground truth DAG.
Revised: Of course in practice one cannot select hyperparameters in this way since we do not have access to the ground truth DAG.

Original: In Appendix A.9, we explain how one could use a held-out data set to select the hyperparameters of score-based approaches and report the results of such a procedure on almost settings presented in this paper.
Revised: In Appendix A.9, we explain how one could use a held-out data set to select the hyperparameters of score-based approaches and report the results of such a procedure on almost settings presented in this paper.

Original: For NOTEARS, DAG-GNN, and GSF, we used the default hyperparameters found in the authors code.
Revised: For NOTEARS, DAG-GNN, and GSF, we used the default hyperparameters found in the authors code.

Original: It (rarely) happens that NOTEARS and DAG-GNN returns a cyclic graph.
Revised: It (rarely) happens that NOTEARS and DAG-GNN returns a cyclic graph.

Original: In those cases, we removed edges starting from the weaker ones to the strongest (according to their respective weighted adjacency matrices), stopping as soon as acyclicity is achieved (similarly to what was explained in
Revised: In those cases, we removed edges starting from the weaker ones to the strongest (according to their respective weighted adjacency matrices), stopping as soon as acyclicity is achieved (similarly to what was explained in

Original: Appendix A.2 for GraN-DAG).
Revised: Appendix A.2 for GraN-DAG).

Original: For GES and PC, we used default hyperparameters of the pcalg
Revised: For GES and PC, we used default hyperparameters of the pcalg

Original: R package.
Revised: R package.

Original: For CAM, we used the the default hyperparameters found in the CAM R package, with default PNS and DAG pruning.
Revised: For CAM, we used the the default hyperparameters found in the CAM R package, with default PNS and DAG pruning.

Original: A. H YPERPARAMETER S ELECTION VIA H ELD - OUT S CORE
Revised: A. H YPERPARAMETER S ELECTION VIA H ELD - OUT S CORE

Original: Most structure/causal learning algorithms have hyperparameters which must be selected prior to learning.
Revised: Most structure/causal learning algorithms have hyperparameters which must be selected prior to learning.

Original: For instance, NOTEARS and GES have a regularizing term in their score controlling the sparsity level of the resulting graph while CAM has a thresholding level for its pruning phase (also controlling the sparsity of the DAG).
Revised: For instance, NOTEARS and GES have a regularizing term in their score controlling the sparsity level of the resulting graph while CAM has a thresholding level for its pruning phase (also controlling the sparsity of the DAG).

Original: GraN-DAG and DAG-GNN have many hyperparameters such as the learning rate and the architecture choice for the neural networks (i.e. number of hidden layers and hidden units per layer).
Revised: GraN-DAG and DAG-GNN have many hyperparameters such as the learning rate and the architecture choice for the neural networks (i.e. number of hidden layers and hidden units per layer).

Original: One approach to selecting hyperparameters in practice consist in trying multiple hyperparameter combinations and keeping the one yielding the best score evaluated on a held-out set (Koller & Friedman, 2009, p. 960).
Revised: One approach to selecting hyperparameters in practice consists in trying multiple hyperparameter combinations and keeping the one yielding the best score evaluated on a held-out set (Koller & Friedman, 2009, p. 960).

Original: By doing so, one can hopefully avoid ﬁnding a DAG which is too dense or too sparse since if the estimated graph contains many spurious edges, the score on the held-out data set should be penalized.
Revised: By doing so, one can hopefully avoid ﬁnding a DAG which is too dense or too sparse since if the estimated graph contains many spurious edges, the score on the held-out data set should be penalized.

Original: In the section, we experiment with this approach on almost all settings and all methods covered in the present paper.
Revised: In the section, we experiment with this approach on almost all settings and all methods covered in the present paper.

Original: Experiments: We explored multiple hyperparameter combinations using random search (Bergstra & Bengio, 2012).
Revised: Experiments: We explored multiple hyperparameter combinations using random search (Bergstra & Bengio, 2012).

Original: Table 15 to Table 23 report results for each dataset types.
Revised: Table 15 to Table 23 report results for each dataset types.

Original: Each table reports the
Revised: Each table reports the

Original: SHD and SID averaged over 10 data sets and for each data set, we tried 50 hyperparameter combinations sampled randomly (see Table 24 for sampling schemes).
Revised: SHD and SID averaged over 10 data sets and for each data set, we tried 50 hyperparameter combinations sampled randomly (see Table 24 for sampling schemes).

Original: The hyperparameter combination yielding the best held-out score among all 50 runs is selected per data set (i.e. the average of SHD and SID scores correspond to potentially different hyperparameter combinations on different data sets).
Revised: The hyperparameter combination yielding the best held-out score among all 50 runs is selected per data set (i.e. the average of SHD and SID scores correspond to potentially different hyperparameter combinations on different data sets).

Original: 80% of the data was used for training and 20% was held out (GraN-DAG uses the same data for early stopping and hyperparameter selection).
Revised: 80% of the data was used for training and 20% was held out (GraN-DAG uses the same data for early stopping and hyperparameter selection).

Original: Note that the held-out score is always evaluated without the regularizing term (e.g. the held-out score of NOTEARS was evaluated without its L regularizer).
Revised: Note that the held-out score is always evaluated without the regularizing term (e.g. the held-out score of NOTEARS was evaluated without its L regularizer).

Original: The symbols ++ and + indicate the hyperparameter search improved performance against default hyperparameter runs above one standard deviation and within one standard deviation, respectively.
Revised: The symbols ++ and + indicate the hyperparameter search improved performance against default hyperparameter runs above one standard deviation and within one standard deviation, respectively.

Original: Analogously for −− and − which indicate a performance reduction.
Revised: Analogously for −− and − which indicate a performance reduction.

Original: The ﬂag ∗∗∗ indicate that, on average, less than 10 hyperparameter combinations among the 50 tried allowed the method toconverge in less than 12 hours.
Revised: The ﬂag ∗∗∗ indicate that, on average, less than 10 hyperparameter combinations among the 50 tried allowed the method to

Original: Analogously, ∗∗ indicates between 10 and 25 runs converged and ∗ indicates between 25 and 45 runs converged.

Original: Discussion: GraN-DAG and DAG-GNN are the methods beneﬁting the most from the hyperparameter selection procedure (although rarely signiﬁcantly).
Revised: Discussion: GraN-DAG and DAG-GNN are the methods beneﬁting the most from the hyperparameter selection procedure (although rarely signiﬁcantly).

Original: This might be explained by the fact that neural networks are in general very sensitive to the choice of hyperparameters.
Revised: This might be explained by the fact that neural networks are in general very sensitive to the choice of hyperparameters.

Original: However, not all methods improved their performance and no method improves its performance in all settings.
Revised: However, not all methods improved their performance and no method improves its performance in all settings.

Original: GES and GSF for instance, often have signiﬁcantly worse results.
Revised: GES and GSF for instance, often have signiﬁcantly worse results.

Original: This might be due to some degree of model misspeciﬁcation which renders the held-out score a poor proxy for graph quality.
Revised: This might be due to some degree of model misspeciﬁcation which renders the held-out score a poor proxy for graph quality.

Original: Moreover, for some methods the gain from the hyperparameter tuning might be outweighed by the loss due to the 20% reduction in training samples.
Revised: Moreover, for some methods the gain from the hyperparameter tuning might be outweighed by the loss due to the 20% reduction in training samples.

Original: Additional implementation details for held-out score evaluation: GraN-DAG makes use of a ﬁnal pruning step to remove spurious edges.
Revised: Additional implementation details for held-out score evaluation: GraN-DAG makes use of a ﬁnal pruning step to remove spurious edges.

Original: One could simply mask the inputs of the NN corresponding to removed edges and evaluate the held-out score.
Revised: One could simply mask the inputs of the NN corresponding to removed edges and evaluate the held-out score.

Original: However, doing so yields an unrepresentative score since some masked inputs have an important role in the learned function and once these inputs are masked, the quality of the ﬁt might greatly suffer.
Revised: However, doing so yields an unrepresentative score since some masked inputs have an important role in the learned function and once these inputs are masked, the quality of the ﬁt might greatly suffer.

Original: To avoid this, we retrained the whole model from scratch on the training set with the masking ﬁxed to the one recovered after pruning.
Revised: To avoid this, we retrained the whole model from scratch on the training set with the masking ﬁxed to the one recovered after pruning.

Original: Then, we evaluate the held-out score with this retrained architecture.
Revised: Then, we evaluate the held-out score with this retrained architecture.

Original: During this retraining phase, the estimated graph is ﬁxed, only the conditional densities are relearned.
Revised: During this retraining phase, the estimated graph is ﬁxed, only the conditional densities are relearned.

Original: Since NOTEARS and DAG-GNN are not always guaranteed to return a DAG (although they almost always do), some extra thresholding might be needed as mentioned in Appendix A.8.
Revised: Since NOTEARS and DAG-GNN are not always guaranteed to return a DAG (although they almost always do), some extra thresholding might be needed as mentioned in Appendix A.8.

Original: Similarly to GraN-DAG’s pruning phase, this step can seriously reduce the quality of the ﬁt.
Revised: Similarly to GraN-DAG’s pruning phase, this step can seriously reduce the quality of the ﬁt.

Original: To avoid this, we also perform a retraining phase for NOTEARS and DAG-GNN.
Revised: To avoid this, we also perform a retraining phase for NOTEARS and DAG-GNN.

Original: The model of CAM is also retrained after its pruning phase prior to evaluating its held-out score
Revised: The model of CAM is also retrained after its pruning phase prior to evaluating its held-out score.

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Figure] [Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] − [Table]
Revised: [Table]

Original: −

Original: [Table]
Revised: [Table]

Original: − 1 } #

Original: [Table]
Revised: [Table]

Original: U { [Table] [Table] [Table]
Revised: [Table]

Original: U {

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Log
Revised: Log

Original: [Table]
Revised: [Table]

Original: [Table]

Original:  U { [Table] [Table] [Table]
Revised: ) ∼ U [ − 2 , − 3] (ﬁrst subproblem) Log(learning rate) ∼ U [ − 3 , − 4] (other subproblems)

Original: [Table]

Original: Bottleneck

Original: [Table]
Revised: [Table]

Original:  U{
Revised: ∼ U

Original: [Table]
Revised: [Table]

Original: NOTEARS
Revised: NOTEARS

Original: [Table]
Revised: [Table]