Original: Scalable Causal Discovery with Score Matching
Revised: Scalable Causal Discovery with Score Matching

Original: Abstract
Revised: Abstract

Original: This paper demonstrates how to discover the whole causal graph from the second derivative of the log-likelihood in observational non-linear additive Gaussian noise models.
Revised: This paper demonstrates how to discover the whole causal graph from the second derivative of the log-likelihood in observational non-linear additive Gaussian noise models.

Original: Leveraging scalable machine learning approaches to approximate the score function ∇ log p ( X ) , we extend the work of Rolland et al.
Revised: Leveraging scalable machine learning approaches to approximate the score function ∇ log p ( X ) , we extend the work of Rolland et al.

Original: [1] that only recovers the topological order from the score and requires an expensive pruning step to discover the edges.
Revised: [1] that only recovers the topological order from the score and requires an expensive pruning step to discover the edges.

Original: Our analysis leads to DAS, a practical algorithm that reduces the complexity of the pruning by a factor proportional to the graph size.
Revised: Our analysis leads to DAS, a practical algorithm that reduces the complexity of the pruning by a factor proportional to the graph size.

Original: In practice, DAS achieves competitive accuracy with current state-of-the-art while being over an order of magnitude faster.
Revised: In practice, DAS achieves competitive accuracy with current state-of-the-art while being over an order of magnitude faster.

Original: Overall, our approach enables principled and scalable causal discovery, significantly lowering the compute bar.
Revised: Overall, our approach enables principled and scalable causal discovery, significantly lowering the compute bar.

Original:  Introduction
Revised:  Introduction

Original: Causal discovery from observational data is a central problem affecting virtually all scientific domains, such as biology, genetics, economics and machine learning [2, 3, 4, 5].
Revised: Causal discovery from observational data is a central problem affecting virtually all scientific domains, such as biology, genetics, economics and machine learning [2, 3, 4, 5].

Original: Given a causal model, under suitable assumptions one can predict the effect of interventions on the system’s variables having access only to observational data.
Revised: Given a causal model, under suitable assumptions one can predict the effect of interventions on the system’s variables having access only to observational data.

Original: In traditional causality research, algorithms to discover causal relationships from observations can be divided in three classes [6, 7].
Revised: In traditional causality research, algorithms to discover causal relationships from observations can be divided in three classes [6, 7].

Original: Constraint based approaches like PC [8], FCI and SGS [9] test conditional independence between the variables, which is notoriously difficult [10].
Revised: Constraint based approaches like PC [8], FCI and SGS [9] test conditional independence between the variables, which is notoriously difficult [10].

Original: Score-based methods like GES [11] define a suitable score function and search for the graph that best fits the data.
Revised: Score-based methods like GES [11] define a suitable score function and search for the graph that best fits the data.

Original: Usually these classes of approaches do not output a unique graph but an equivalence class.
Revised: Usually these classes of approaches do not output a unique graph but an equivalence class.

Original: Finally, a restricted model class assumption, e g. non-linear relations and additive
Revised: Finally, a restricted model class assumption, e g. non-linear relations and additive

Original: Gaussian noise, allows to identify the Directed Acyclic Graph (DAG)
Revised: Gaussian noise, allows to identify the Directed Acyclic Graph (DAG)

Original: underlying the observations [5, 12, 13, 14, 15].
Revised: underlying the observations [5, 12, 13, 14, 15].

Original: The main challenge is that enforcing the DAG constraint has a cubic per-iteration cost in the number of variables, making the optimization the computational bottleneck.
Revised: The main challenge is that enforcing the DAG constraint has a cubic per-iteration cost in the number of variables, making the optimization the computational bottleneck.

Original: A step towards better scalability is Rolland et al.
Revised: A step towards better scalability is Rolland et al.

Original: [1] that recently proposed the SCORE algorithm: first they efficiently estimate the score function ∇ log p ( X ) , then they recover the topological order from the Jacobian of the score, and finally prune the fully connected DAG by the method proposed in
Revised: [1] that recently proposed the SCORE algorithm: first they efficiently estimate the score function ∇ log p ( X ) , then they recover the topological order from the Jacobian of the score, and finally prune the fully connected DAG by the method proposed in

Original: CAM [12].
Revised: CAM [12].

Original: The pruning step is the bottleneck of SCORE, scaling cubic in the number of nodes.
Revised: The pruning step is the bottleneck of SCORE, scaling cubic in the number of nodes.

Original: In this work, we show that the Jacobian of the score allows to recover both the skeleton and the direction of the edges in the causal DAG.
Revised: In this work, we show that the Jacobian of the score allows to recover both the skeleton and the direction of the edges in the causal DAG.

Original: Theoretically, this implies that we can get rid of the costly pruning step in SCORE [1] as all information about the causal structure is already contained in the Jacobian of the score.
Revised: Theoretically, this implies that we can get rid of the costly pruning step in SCORE [1] as all information about the causal structure is already contained in the Jacobian of the score.

Original: While our analysis yields a practical algorithm, we found it beneficial to first identify few candidate edges and still retain a final cheap pruning step.
Revised: While our analysis yields a practical algorithm, we found it beneficial to first identify few candidate edges and still retain a final cheap pruning step.

Original: This is now much more efficient as most of the edges have already been detected and it is only needed to correct mistakes from the finite samples approximation of the score, reducing the complexity by a factor proportional to the number of nodes in the graph.
Revised: This is now much more efficient as most of the edges have already been detected and it is only needed to correct mistakes from the finite samples approximation of the score, reducing the complexity by a factor proportional to the number of nodes in the graph.

Original: [Figure]
Revised: [Figure]

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We demonstrate how to theoretically recover the full causal DAG from the score of the data distribution.
Revised: • We demonstrate how to theoretically recover the full causal DAG from the score of the data distribution.

Original: This extends prior work showing that the topological order can be recovered from the score [1].
Revised: This extends prior work showing that the topological order can be recovered from the score [1].

Original: • We introduce DAS (acronym for Discovery At Scale), an algorithm for efficient and scalable causal discovery.
Revised: • We introduce DAS (acronym for Discovery At Scale), an algorithm for efficient and scalable causal discovery.

Original: While our approach is marginally less accurate than the state of the art ([1], [12], [13]), it improves the runtime by at least an order of magnitude in the graph size, as shown in Figure 1.
Revised: While our approach is marginally less accurate than the state of the art ([1], [12], [13]), it improves the runtime by at least an order of magnitude in the graph size, as shown in Figure 1.

Original: We demonstrate the speedup improvement on synthetic graphs with up to a thousand nodes.
Revised: We demonstrate the speedup improvement on synthetic graphs with up to a thousand nodes.

Original:  Background knowledge
Revised:  Background knowledge

Original: We introduce the background needed for our analysis starting from the formalism of structural causal models.
Revised: We introduce the background needed for our analysis starting from the formalism of structural causal models.

Original: Structural Causal Models One way to formalize causal relationships between variables is with an additive Structural Causal Model (SCM).
Revised: Structural Causal Models One way to formalize causal relationships between variables is with an additive Structural Causal Model (SCM).

Original: Consider a set X
Revised: Consider a set X

Original: = { X i } di =1 of observable vertices of a
Revised: = { X i } di =1 of observable vertices of a

Original: DAG.
Revised: DAG.

Original: We assume that the structure of the graph can be expressed in the functional relationship
Revised: We assume that the structure of the graph can be expressed in the functional relationship

Original: [Equation]
Revised: [Equation]

Original: with pa i ( X ) set of parent nodes of X i in the directed network.
Revised: with pa i ( X ) set of parent nodes of X i in the directed network.

Original: We will assume X i ∈ R , additive and independently drawn Gaussian noise elements
Revised: We will assume X i ∈ R , additive and independently drawn Gaussian noise elements

Original: ϵ i , as well as f i to be twice continuously differentiable and non-linear in every component.
Revised: ϵ i , as well as f i to be twice continuously differentiable and non-linear in every component.

Original: Additionally we assume that f i restricted to any interval is still non-linear.
Revised: Additionally we assume that f i restricted to any interval is still non-linear.

Original: Recursive application of (1) allows to derive the joint probability distribution p ( X 1 , X 2 , . .
Revised: Recursive application of (1) allows to derive the joint probability distribution p ( X 1 , X 2 , . .

Original: X d ) .
Revised: X d ) .

Original: As this probability is over vertices of a directed acyclic graph, the following factorization holds [4, 16]:
Revised: As this probability is over vertices of a directed acyclic graph, the following factorization holds [4, 16]:

Original: [Equation]
Revised: [Equation]

Original: Usually the form of the f i in the model (1) is not known and neither is the probability in (2), while we can only access a set of observations from the joint distribution.
Revised: Usually the form of the f i in the model (1) is not known and neither is the probability in (2), while we can only access a set of observations from the joint distribution.

Original: Given these observations the task is to identify the causal structure of the graph underlying the SCM.
Revised: Given these observations the task is to identify the causal structure of the graph underlying the SCM.

Original: This problem is known as causal discovery.
Revised: This problem is known as causal discovery.

Original: As mentioned before one solution is to use data to estimate a topological ordering of the variables in X , and then to choose edges of the DAG between those admitted by such sorting.
Revised: As mentioned before one solution is to use data to estimate a topological ordering of the variables in X , and then to choose edges of the DAG between those admitted by such sorting.

Original: In our approach we select edges that satisfy constraints derived from the score function.
Revised: In our approach we select edges that satisfy constraints derived from the score function.

Original: Topological ordering definition Let G =
Revised: Topological ordering definition Let G =

Original: ( X , E ) be a DAG.
Revised: ( X , E ) be a DAG.

Original: An ordering of the nodes
Revised: An ordering of the nodes

Original: X π = X π 1 , . . .
Revised: X π = X π 1 , . . .

Original: , X π d is a topological ordering relative to G if, whenever we have X π i → X π j ∈ E , then i < j [17].
Revised: , X π d is a topological ordering relative to G if, whenever we have X π i → X π j ∈ E , then i < j [17].

Original:  Deducing causal structure from the score
Revised:  Deducing causal structure from the score

Original: For the causal discovery problem under analysis we consider an observable X ∈ R d whose entries
Revised: For the causal discovery problem under analysis we consider an observable X ∈ R d whose entries

Original: X i are vertices of a graph generated according to the model in (1).
Revised: X i are vertices of a graph generated according to the model in (1).

Original: In the next section, we show how the score function is in principle sufficient to solve this task.
Revised: In the next section, we show how the score function is in principle sufficient to solve this task.

Original: In particular we discuss the key concepts introduced in [1], and we slightly revisit their Lemma 1 to give a more precise statement that holds almost surely and avoids some (trivial) corner cases.
Revised: In particular we discuss the key concepts introduced in [1], and we slightly revisit their Lemma 1 to give a more precise statement that holds almost surely and avoids some (trivial) corner cases.

Original: 3.1 Deriving constraints
Revised: 3.1 Deriving constraints

Original: Authors of [1] show how to efficiently estimate the score function s ( X )
Revised: Authors of [1] show how to efficiently estimate the score function s ( X )

Original: = ∇ log p
Revised: = ∇ log p

Original: ( X ) and its
Revised: ( X ) and its

Original: Jacobian exploiting the Stein identity.
Revised: Jacobian exploiting the Stein identity.

Original: Then they propose a method to identify leaf nodes in a causal graph generated according to (1) by inspection of the diagonal elements of the Jacobian of the score.
Revised: Then they propose a method to identify leaf nodes in a causal graph generated according to (1) by inspection of the diagonal elements of the Jacobian of the score.

Original: In the following part we show how additional constraints on the off-diagonal elements of the Jacobian matrix itself can be defined to identify directed edges in the graph.
Revised: In the following part we show how additional constraints on the off-diagonal elements of the Jacobian matrix itself can be defined to identify directed edges in the graph.

Original: Starting from Equation (2) we can derive s ( X ) in closed form from log p ( X ) as follows:
Revised: Starting from Equation (2) we can derive s ( X ) in closed form from log p ( X ) as follows:

Original: [Equation]
Revised: [Equation]

Original: The j -th entry of ∇ log p ( X ) therefore is
Revised: The j -th entry of ∇ log p ( X ) therefore is

Original: [Equation]
Revised: [Equation]

Original: We observe that for a leaf node l , X l ∈ X , the partial derivative of (4) over X j with j ̸ = l is:
Revised: We observe that for a leaf node l , X l ∈ X , the partial derivative of (4) over X j with j ̸ = l is:

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: else .
Revised: else .

Original: [Equation]
Revised: [Equation]

Original: It is worth to notice that 1 σ 2 l ∂f l ∂X j (pa l ( X )) might still be vanishing for some values of pa l ( X ) even if X j ∈ pa l ( X ) , for instance if the function has a maximum or a minimum: given the assumption on f l non-linear even when considered on a restricted interval, these events happen with probability zero, such that 1 σ 2 l ∂f l ∂X j (pa l ( X ))
Revised: It is worth to notice that 1 σ 2 l ∂f l ∂X j (pa l ( X )) might still be vanishing for some values of pa l ( X ) even if X j ∈ pa l ( X ) , for instance if the function has a maximum or a minimum: given the assumption on f l non-linear even when considered on a restricted interval, these events happen with probability zero, such that 1 σ 2 l ∂f l ∂X j (pa l ( X ))

Original: ̸ = 0 holds almost surely .
Revised: ̸ = 0 holds almost surely .

Original: We prove that the condition in Equation (5) allows to derive a criterion to identify parents of a given leaf node.
Revised: We prove that the condition in Equation (5) allows to derive a criterion to identify parents of a given leaf node.

Original: Lemma 1 (Adapted from [1]) .
Revised: Lemma 1 (Adapted from [1]) .

Original: Let p be the probability density function of a random variable
Revised: Let p be the probability density function of a random variable

Original: X ∈ R d defined via non-linear additive Gaussian noise model (1) .
Revised: X ∈ R d defined via non-linear additive Gaussian noise model (1) .

Original: Let also s ( X ) =
Revised: Let also s ( X ) =

Original: ∇ log p
Revised: ∇ log p

Original: ( X ) be the associated score function.
Revised: ( X ) be the associated score function.

Original: Without loss of generality, assume a topological ordering X π
Revised: Without loss of generality, assume a topological ordering X π

Original: =
Revised: =

Original: ( X 1 , . . .
Revised: ( X 1 , . . .

Original: , X d ) .
Revised: , X d ) .

Original: Then given a leaf l :
Revised: Then given a leaf l :

Original: [Equation]
Revised: [Equation]

Original: The proof is provided in the Appendix A.2.
Revised: The proof is provided in the Appendix A.2.

Original: Novelties of Lemma 1 We now provide a discussion of the key differences of our Lemma 1 with
Revised: Novelties of Lemma 1 We now provide a discussion of the key differences of our Lemma 1 with

Original: Lemma 1 in [1].
Revised: Lemma 1 in [1].

Original: Their formulation requires Var (cid:104) 1 σ 2 l ∂f l ∂X j (pa l ( X ))
Revised: Their formulation requires Var (cid:104) 1 σ 2 l ∂f l ∂X j (pa l ( X ))

Original: (cid:105) ̸ = 0 ⇔ X j ∈ pa l , where X l is a leaf node.
Revised: (cid:105) ̸ = 0 ⇔ X j ∈ pa l , where X l is a leaf node.

Original: We illustrate the problem with this considering a simple two variables case with graph
Revised: We illustrate the problem with this considering a simple two variables case with graph

Original: X 1 −→ X 2 : if parent node X 1 has zero variance, their selection condition would brake, predicting a graph with X 1 and X 2 independent.
Revised: X 1 −→ X 2 : if parent node X 1 has zero variance, their selection condition would brake, predicting a graph with X 1 and X 2 independent.

Original: While this case would be ruled out by the assumption of variance larger than zero for every node, in practice this can be a problem.
Revised: While this case would be ruled out by the assumption of variance larger than zero for every node, in practice this can be a problem.

Original: Given a finite sample
Revised: Given a finite sample

Original: X ∈ R n × d and its topological ordering X π , if parents of a leaf X l show small variance in the sample, we might still mistake the oscillation observed in 1 σ 2 l ∂f l ∂X j (pa l ( X )) for statistical error due to finite set estimates, discarding an existing edge.
Revised: X ∈ R n × d and its topological ordering X π , if parents of a leaf X l show small variance in the sample, we might still mistake the oscillation observed in 1 σ 2 l ∂f l ∂X j (pa l ( X )) for statistical error due to finite set estimates, discarding an existing edge.

Original: In practice, we rely on the sample mean of the absolute value of the Jacobian entries 1 σ 2 l ∂f l ∂X j (pa l ( X )) for the implementation of Lemma 1: this estimator is potentially subject to the same issues, but shows better robustness properties than the sample variance (due to the absolute value) and is a lower moment thus yielding lower error (estimating variance requires estimating the mean first, such that any statistical error in the mean estimator affects the variance estimator), making it a preferable choice.
Revised: In practice, we rely on the sample mean of the absolute value of the Jacobian entries 1 σ 2 l ∂f l ∂X j (pa l ( X )) for the implementation of Lemma 1: this estimator is potentially subject to the same issues, but shows better robustness properties than the sample variance (due to the absolute value) and is a lower moment thus yielding lower error (estimating variance requires estimating the mean first, such that any statistical error in the mean estimator affects the variance estimator), making it a preferable choice.

Original: In practice we can exploit Equation (5) to reconstruct the entire graph only if a sorting X π is provided.
Revised: In practice we can exploit Equation (5) to reconstruct the entire graph only if a sorting X π is provided.

Original: To see why, consider the last entry X l of X π : by definition of topological ordering X l is a leaf.
Revised: To see why, consider the last entry X l of X π : by definition of topological ordering X l is a leaf.

Original: Then we can apply Lemma 1 doing partial derivatives of s l ( X ) over all nodes in X π \ { X l } and identify as parents those that satisfy the required constraint.
Revised: Then we can apply Lemma 1 doing partial derivatives of s l ( X ) over all nodes in X π \ { X l } and identify as parents those that satisfy the required constraint.

Original: At this point, we remove X l from the ordering
Revised: At this point, we remove X l from the ordering

Original: X π and repeat the procedure on the pruned graph with vertices X \ X l .
Revised: X π and repeat the procedure on the pruned graph with vertices X \ X l .

Original: By iterating these steps over each node in the ordering from last to source we can identify the exact graph.
Revised: By iterating these steps over each node in the ordering from last to source we can identify the exact graph.

Original: Next we derive a causal discovery algorithm based on this approach, and show how it retains performance with respect to other state of the art methods, while scaling better in the number of nodes.
Revised: Next we derive a causal discovery algorithm based on this approach, and show how it retains performance with respect to other state of the art methods, while scaling better in the number of nodes.

Original: 3.2 DAS: an algorithm for causal Discovery At Scale
Revised: 3.2 DAS: an algorithm for causal Discovery At Scale

Original: We want to use the constraint of Lemma 1 on the score function to derive an algorithm for causal discovery which is faster and exhibits better scaling properties in the number of nodes than any other technique to our knowledge.
Revised: We want to use the constraint of Lemma 1 on the score function to derive an algorithm for causal discovery which is faster and exhibits better scaling properties in the number of nodes than any other technique to our knowledge.

Original: In practice, we rely on SCORE for estimating both X π and the Jacobian matrix J ( s ( X )) from a set of n observations, generated according to the model assumptions in
Revised: In practice, we rely on SCORE for estimating both X π and the Jacobian matrix J ( s ( X )) from a set of n observations, generated according to the model assumptions in

Original: Equation (1).
Revised: Equation (1).

Original: Given a topological ordering, we filter edges for each node by inspecting the averages of the absolute values of the off-diagonal elements of the estimator ˆ J ( s ( X )) according to the criterion of Lemma 1.
Revised: Given a topological ordering, we filter edges for each node by inspecting the averages of the absolute values of the off-diagonal elements of the estimator ˆ J ( s ( X )) according to the criterion of Lemma 1.

Original: Finally, we obtain the output graph by running CAM pruning method on the resulting adjacency matrix, reducing the number of false positives.
Revised: Finally, we obtain the output graph by running CAM pruning method on the resulting adjacency matrix, reducing the number of false positives.

Original: Note that since we use an approximation of J ( s ( X ))
Revised: Note that since we use an approximation of J ( s ( X ))

Original: its entries are never precisely equal to zero.
Revised: its entries are never precisely equal to zero.

Original: According to Lemma 1, we consider the absolute value of the l -th row of the n Jacobian matrices, and look for entries with non-zero mean: this can be achieved by statistical hypothesis testing, where the idea is to test for the mean of a sample to be different from zero.
Revised: According to Lemma 1, we consider the absolute value of the l -th row of the n Jacobian matrices, and look for entries with non-zero mean: this can be achieved by statistical hypothesis testing, where the idea is to test for the mean of a sample to be different from zero.

Original: In practice, for each off-diagonal entries J l,j of row l we test
Revised: In practice, for each off-diagonal entries J l,j of row l we test

Original: H 0 : E [ | J l,j | ] = 

Revised: H 0 : E [ | J l,j | ] =

Original: and the alternative H 1 : E [ | J l,j | ] > 0 , with X j potential parent of X l .
Revised: and the alternative H 1 : E [ | J l,j | ] > 0 , with X j potential parent of X l .

Original: If we reject the null with p-value = 0 .
Revised: If we reject the null with p-value = 0 .

Original: 01 , then X j is added to the parents of X l .
Revised: 01 , then X j is added to the parents of X l .

Original: Experiments Figure 2 summarizes experimental results of DAS in comparison with other state of the art algorithms for causal discovery on non-linear Gaussian additive noise models: in particular we select CAM [12], SCORE [1] and GranDAG [13] as benchmarks.
Revised: Experiments Figure 2 summarizes experimental results of DAS in comparison with other state of the art algorithms for causal discovery on non-linear Gaussian additive noise models: in particular we select CAM [12], SCORE [1] and GranDAG [13] as benchmarks.

Original: The causal graphs are synthetically generated using the Erdös-Renyi model [18].
Revised: The causal graphs are synthetically generated using the Erdös-Renyi model [18].

Original: We run experiments fixing the number of nodes d as well as the sparsity of the graph by setting the expected amount of edges to be equal to d (ER1) or 4 d (ER4).
Revised: We run experiments fixing the number of nodes d as well as the sparsity of the graph by setting the expected amount of edges to be equal to d (ER1) or 4 d (ER4).

Original: Figure 2 illustrates how performance is retained with respect to the competitors, while Figure 1 shows how this results are achieved with significantly lower computational time.
Revised: Figure 2 illustrates how performance is retained with respect to the competitors, while Figure 1 shows how this results are achieved with significantly lower computational time.

Original: A comprehensive presentation of all the experimental results is provided in the Appendix A.5.
Revised: A comprehensive presentation of all the experimental results is provided in the Appendix A.5.

Original: [Figure]
Revised: [Figure]

Original: Algorithmic complexity Considering an input matrix n × d with n the number of samples and d the number of nodes, the overall complexity of DAS is O ( dn 3 + d 2 ) .
Revised: Algorithmic complexity Considering an input matrix n × d with n the number of samples and d the number of nodes, the overall complexity of DAS is O ( dn 3 + d 2 ) .

Original: Indeed estimating the topological sorting with SCORE involves inverting d times a n × n matrix, with a complexity O ( dn 3 ) .
Revised: Indeed estimating the topological sorting with SCORE involves inverting d times a n × n matrix, with a complexity O ( dn 3 ) .

Original: Additionally the edge search step of DAS requires iterating over the d elements of the ordering, each time inspecting a list of size ≤ d
Revised: Additionally the edge search step of DAS requires iterating over the d elements of the ordering, each time inspecting a list of size ≤ d

Original: (see Algorithm 1) yielding a O ( d 2 ) contribution.
Revised: (see Algorithm 1) yielding a O ( d 2 ) contribution.

Original: On the other hand the bottleneck of SCORE, arguably the most scalable state-of-the-art algorithm for causal discovery, is the first step in the pruning approach, namely Preliminary Neighbours Search (PNS): this procedure acts as an edge selection preliminary to CAM pruning, amounting to complexity O ( nd 3 ) .
Revised: On the other hand the bottleneck of SCORE, arguably the most scalable state-of-the-art algorithm for causal discovery, is the first step in the pruning approach, namely Preliminary Neighbours Search (PNS): this procedure acts as an edge selection preliminary to CAM pruning, amounting to complexity O ( nd 3 ) .

Original: Therefore our use of the score for preliminary edges selection in place of PNS dramatically improves the execution time allowing to scale causal discovery in high dimensions by a factor of O ( d ) .
Revised: Therefore our use of the score for preliminary edges selection in place of PNS dramatically improves the execution time allowing to scale causal discovery in high dimensions by a factor of O ( d ) .

Original: Acknowledgements
Revised: Acknowledgements

Original: We want to thank Volkan Cevher for the valuable discussions.
Revised: We want to thank Volkan Cevher for the valuable discussions.

Original: This work has been carried out at the Machine Learning Genoa (MaLGa) center, Università di Genova (IT).
Revised: This work has been carried out at the Machine Learning Genoa (MaLGa) center, Università di Genova (IT).

Original: It has been supported by
Revised: It has been supported by

Original: AFOSR, grant n. FA8655-20-1-7035.
Revised: AFOSR, grant n. FA8655-20-1-7035.

Original: Francesco Montagna is supported by Programma Operativo Nazionale ricerca e innovazione 2014-2020 .
Revised: FM is supported by Programma Operativo Nazionale ricerca e innovazione 2014-2020 .

Original: A Appendix A.1 DAS pseudo-code
Revised: A Appendix A.1 DAS pseudo-code

Original: The pseudo-code of Algorithm 1 provides an overview of the implementation details of DAS.
Revised: The pseudo-code of Algorithm 1 provides an overview of the implementation details of DAS.

Original: Algorithm 1 DAS
Revised: Algorithm 1 DAS

Original: Input: data matrix X ∈ R n × d
Revised: Input: data matrix X ∈ R n × d

Original: X π ← SCORE( X ) (such that X π [ d ] source node)
Revised: X π ← SCORE( X ) (such that X π [ d ] source node)

Original: A ← d × d zeros adjacency matrix remaining nodes = 1 , . .
Revised: A ← d × d zeros adjacency matrix remaining nodes = 1 , . .

Original: , d δ = 0 .
Revised: , d δ = 0 .

Original: 01 (hyperparameter) for X l in X π
Revised: 01 (hyperparameter) for X l in X π

Original: do ˆ J ← Average (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ∂X j ( X ) (cid:12)(cid:12)(cid:12)(cid:105) X j ∈ X \{ X l } (estimate from SCORE) threshold =
Revised: do ˆ J ← Average (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ∂X j ( X ) (cid:12)(cid:12)(cid:12)(cid:105) X j ∈ X \{ X l } (estimate from SCORE) threshold =

Original: δ · max( ˆ J ) for j in remaining nodes do if ˆ J [ j ] > threshold then
Revised: δ · max( ˆ J ) for j in remaining nodes do if ˆ J [ j ] > threshold then

Original: A [ j, l ] =end if end for
Revised: A [ j, l ] =end if end for

Original: Remove l -th column from X
Revised: Remove l -th column from X

Original: Remove l from remaining nodes end for
Revised: Remove l from remaining nodes end for

Original: Graph G ← CAM-pruning( A )
Revised: Graph G ← CAM-pruning( A )

Original: A.2 Proof of Lemma
Revised: A.2 Proof of Lemma

Original: In this section we provide a proof of the statement of Lemma 1.
Revised: In this section we provide a proof of the statement of Lemma 1.

Original: Proof.
Revised: Proof.

Original: For a leaf l the score of Equation (4) becomes s l ( X ) =
Revised: For a leaf l the score of Equation (4) becomes s l ( X ) =

Original: − X l − f l (pa l ( X )) σ 2 l .
Revised: − X l − f l (pa l ( X )) σ 2 l .

Original: We compute the partial derivative
Revised: We compute the partial derivative

Original: [Equation]
Revised: [Equation]

Original: and observe that: (i) E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) ̸ = 

Revised: and observe that: (i) E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) ̸ =

Original: ⇒ X j ∈ pa l ( X ) .
Revised: ⇒ X j ∈ pa l ( X ) .

Original: By contradiction, consider X j ̸∈ pa l ( X ) : being f l (pa l ( X )) constant in X j , then ∂f l (pa l ( X ))
Revised: By contradiction, consider X j ̸∈ pa l ( X ) : being f l (pa l ( X )) constant in X j , then ∂f l (pa l ( X ))

Original: ∂X j = 0 for every X ∈ R d by definition of derivative.
Revised: ∂X j = 0 for every X ∈ R d by definition of derivative.

Original: Then, E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X )
Revised: Then, E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X )

Original: ∂X
Revised: ∂X

Original: j (cid:12)(cid:12)(cid:12)(cid:105) = 0 , which contradicts the hypothesis.
Revised: j (cid:12)(cid:12)(cid:12)(cid:105) = 0 , which contradicts the hypothesis.

Original: (ii) X j ∈ pa l ( X ) ⇒ E
Revised: (ii) X j ∈ pa l ( X ) ⇒ E

Original: (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) ̸ = 0 : we observe from Equation (5) that ∂f l ∂X j (pa l ( X )) ̸
Revised: (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) ̸ = 0 : we observe from Equation (5) that ∂f l ∂X j (pa l ( X )) ̸

Original: =almost surely , such that (cid:12) (cid:12)(cid:12)
Revised: =almost surely , such that (cid:12) (cid:12)(cid:12)

Original: ∂f l ∂X j (pa l ( X )) (cid:12)(cid:12)(cid:12) > 0 almost surely .
Revised: ∂f l ∂X j (pa l ( X )) (cid:12)(cid:12)(cid:12) > 0 almost surely .

Original: Being the probability of vanishing (cid:12)(cid:12)(cid:12) ∂f l ∂X j (pa l ( X )) (cid:12)(cid:12)(cid:12) equals to zero, then the expectation E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) is equivalent to the integral (cid:90)
Revised: Being the probability of vanishing (cid:12)(cid:12)(cid:12) ∂f l ∂X j (pa l ( X )) (cid:12)(cid:12)(cid:12) equals to zero, then the expectation E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) is equivalent to the integral (cid:90)

Original: X + (cid:12)(cid:12)(cid:12) ∂f l ∂X j (pa l ( X ))
Revised: X + (cid:12)(cid:12)(cid:12) ∂f l ∂X j (pa l ( X ))

Original: (cid:12)(cid:12)(cid:12) dP ( X ) , with X + ⊆
Revised: (cid:12)(cid:12)(cid:12) dP ( X ) , with X + ⊆

Original: R
Revised: R

Original: d the subset of values where
Revised: d the subset of values where

Original: (cid:12)(cid:12)(cid:12) ∂f l ∂X j (pa l ( X )) (cid:12)(cid:12)(cid:12) is strictly positive.
Revised: (cid:12)(cid:12)(cid:12) ∂f l ∂X j (pa l ( X )) (cid:12)(cid:12)(cid:12) is strictly positive.

Original: Since the integral of a strictly positive function is strictly positive itself, then E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) > 0 .
Revised: Since the integral of a strictly positive function is strictly positive itself, then E (cid:104)(cid:12)(cid:12)(cid:12) ∂s l ( X ) ∂X j (cid:12)(cid:12)(cid:12)(cid:105) > 0 .

Original: A.3 Lemma 1 and Markov networks
Revised: A.3 Lemma 1 and Markov networks

Original: In this section we provide a more detailed analysis about Lemma 1, in particular we want to show how it relates to prior work on Markov networks.
Revised: In this section we provide a more detailed analysis about Lemma 1, in particular we want to show how it relates to prior work on Markov networks.

Original: The findings of Lemma 1 on identification of the causal structure from the score function are not completely surprising in the light of previous results on Markov networks [19, 20].
Revised: The findings of Lemma 1 on identification of the causal structure from the score function are not completely surprising in the light of previous results on Markov networks [19, 20].

Original: Given a collection of random variables X
Revised: Given a collection of random variables X

Original: =
Revised: =

Original: ( X 1 , X 2 , . . .
Revised: ( X 1 , X 2 , . . .

Original: , X d ) with joint density p ( X ) , the information of conditional independencies between the variables of X can be embedded in a simple undirected Markov network
Revised: , X d ) with joint density p ( X ) , the information of conditional independencies between the variables of X can be embedded in a simple undirected Markov network

Original: G =
Revised: G =

Original: ( V , E ) , where edges ( i, j ) encode some sort of probabilistic interaction between the pairs of random variables
Revised: ( V , E ) , where edges ( i, j ) encode some sort of probabilistic interaction between the pairs of random variables

Original: X i , X j .
Revised: X i , X j .

Original: In particular Spantini et al.
Revised: In particular Spantini et al.

Original: [19] proved how to construct a Markov graph reading the conditional independence of pairs of random variables as follow:
Revised: [19] proved how to construct a Markov graph reading the conditional independence of pairs of random variables as follow:

Original: [Equation]
Revised: [Equation]

Original: where ∂ ij ( · ) denotes the ij -th mixed partial derivative and ∂ ij log p
Revised: where ∂ ij ( · ) denotes the ij -th mixed partial derivative and ∂ ij log p

Original: ( X ) is an entry of the Jacobian of the score.
Revised: ( X ) is an entry of the Jacobian of the score.

Original: By adding edges between each couple of nodes that appears not to satisfy Equation (7), we obtain an undirected graph encoding all and only the existing conditional independencies between the variables of X .
Revised: By adding edges between each couple of nodes that appears not to satisfy Equation (7), we obtain an undirected graph encoding all and only the existing conditional independencies between the variables of X .

Original: Equation (5) of our work discovers the same constraint in a slightly different setting: rather than evaluating ∂ ij log p ( X ) for each node against every other, we follow an iterative approach where first we identify a leaf X l and then we test its mixed derivatives only against nodes coming before in the topological ordering.
Revised: Equation (5) of our work discovers the same constraint in a slightly different setting: rather than evaluating ∂ ij log p ( X ) for each node against every other, we follow an iterative approach where first we identify a leaf X l and then we test its mixed derivatives only against nodes coming before in the topological ordering.

Original: By the time we find an edge we know its direction as we know that X l is a leaf, which breaks the symmetry in the relation.
Revised: By the time we find an edge we know its direction as we know that X l is a leaf, which breaks the symmetry in the relation.

Original: The additional pieces of information we have access to - namely that X l has no children in the considered graph and that the sorting is given - allows to identify the direction of a detected link, since a leaf can only be the effect in a causal relation.
Revised: The additional pieces of information we have access to - namely that X l has no children in the considered graph and that the sorting is given - allows to identify the direction of a detected link, since a leaf can only be the effect in a causal relation.

Original: Moreover Lemma 1 ensures correct identification of directed v-structures like i → j ← k that instead in the conditional independence map are moralized with an additional link ( i, k ) .
Revised: Moreover Lemma 1 ensures correct identification of directed v-structures like i → j ← k that instead in the conditional independence map are moralized with an additional link ( i, k ) .

Original: A.4 Sachs dataset experiments
Revised: A.4 Sachs dataset experiments

Original: Table 1 reports experimental outcomes on Sachs [2], a real-world dataset with 11 variables popular for causal discovery.
Revised: Table 1 reports experimental outcomes on Sachs [2], a real-world dataset with 11 variables popular for causal discovery.

Original: We see that with δ = 0 .
Revised: We see that with δ = 0 .

Original: 01 our method matches SCORE.
Revised: 01 our method matches SCORE.

Original: [Table]
Revised: [Table]

Original: A.5 Synthetic data experiments
Revised: A.5 Synthetic data experiments

Original: In this section we summarize experimental outcomes of DAS on synthetic data generated with
Revised: In this section we summarize experimental outcomes of DAS on synthetic data generated with

Original: Erdös-Renyi [18] models and on Scale-free graphs.
Revised: Erdös-Renyi [18] models and on Scale-free graphs.

Original: For both type of data we provide experiments on sparser (Table 4 and Table 5) and denser (Table 2 and Table 3) graphs.
Revised: For both type of data we provide experiments on sparser (Table 4 and Table 5) and denser (Table 2 and Table 3) graphs.

Original: The metrics used are precision, recall, Structural Hamming Distance (SHD) – which is computed as the sum of false positive, false negative and wrongly directed edges – and Structural Intervention
Revised: The metrics used are precision, recall, Structural Hamming Distance (SHD) – which is computed as the sum of false positive, false negative and wrongly directed edges – and Structural Intervention

Original: Distance (SID) [21] – accounting for the number of miscalculated interventional distributions that would result from the inferred graph.
Revised: Distance (SID) [21] – accounting for the number of miscalculated interventional distributions that would result from the inferred graph.

Original: From Table 2 we can see that on denser graphs (ER4) our method maintains similar performance with respect to the other three for nodes up to 50, while being considerably faster in particular with respect to GraNDAG and CAM.
Revised: From Table 2 we can see that on denser graphs (ER4) our method maintains similar performance with respect to the other three for nodes up to 50, while being considerably faster in particular with respect to GraNDAG and CAM.

Original: As d increases, the accuracy gap with SCORE reduces up to the point that for 200 nodes we observe better SID for our algorithm.
Revised: As d increases, the accuracy gap with SCORE reduces up to the point that for 200 nodes we observe better SID for our algorithm.

Original: At d ≥ 500 it becomes arguably impossible to run
Revised: At d ≥ 500 it becomes arguably impossible to run

Original: SCORE on a personal computer in a finite amount of time, whereas DAS is the only reasonable option.
Revised: SCORE on a personal computer in a finite amount of time, whereas DAS is the only reasonable option.

Original: Similarly, the performances across the different methods are comparable when running inference on sparser graphs (ER1), as reported in Table 4.
Revised: Similarly, the performances across the different methods are comparable when running inference on sparser graphs (ER1), as reported in Table 4.

Original: These results are directly observable in Figure 2: each algorithm shows a similar degrade in performance with the number of nodes increasing
Revised: These results are directly observable in Figure 2: each algorithm shows a similar degrade in performance with the number of nodes increasing

Original: [Table]
Revised: [Table]

Original: and bars set to close SHD values.
Revised: and bars set to close SHD values.

Original: Nevertheless, in Figure 1 it clearly appears that DAS achieves these metrics in a significantly smaller amount of time, supporting the claim of better efficiency in terms of velocity and scalability of our approach.
Revised: Nevertheless, in Figure 1 it clearly appears that DAS achieves these metrics in a significantly smaller amount of time, supporting the claim of better efficiency in terms of velocity and scalability of our approach.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]