Original: C AUSAL D ISCOVERY VIA C HOLESKY F ACTORIZATION
Revised: C AUSAL D ISCOVERY VIA C HOLESKY F ACTORIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Discovering the causal relationship via recovering the directed acyclic graph ( DAG ) structure from the observed data is a challenging combinatorial problem.
Revised: Discovering the causal relationship via recovering the directed acyclic graph (DAG) structure from the observed data is a challenging combinatorial problem.

Original: This paper proposes an extremely fast, easy to implement, and high-performance
Revised: This paper proposes an extremely fast, easy to implement, and high-performance

Original: DAG structure recovering algorithm.
Revised: DAG structure recovering algorithm.

Original: The algorithm is based on the Cholesky factorization of the covariance/precision matrix.
Revised: The algorithm is based on the Cholesky factorization of the covariance/precision matrix.

Original: The time complexity of the algorithm is O
Revised: The time complexity of the algorithm is O

Original: ( p 2 n + p 3 ) , where p and n are the numbers of nodes and samples, respectively.
Revised: ( p 2 n + p 3 ) , where p and n are the numbers of nodes and samples, respectively.

Original: Under proper assumptions, we show that our algorithm takes
Revised: Under proper assumptions, we show that our algorithm takes

Original: O (log( p/(cid:15) )) samples to exactly recover the DAG structure with probability at least 1 − (cid:15) .
Revised: O (log( p/(cid:15) ) R ) samples to exactly recover the DAG structure with probability at least 1 − (cid:15) where R is the bound of l 2 norm of the observed random vector.

Original: In both time and sample complexities, our algorithm is better than previous algorithms.
Revised: In both time and sample complexities, our algorithm is better than previous algorithms.

Original: On synthetic and real-world data sets, our algorithm is signiﬁcantly faster than previous methods and achieves state-of-the-art performance.
Revised: On synthetic and real-world data sets, our algorithm is signiﬁcantly faster than previous methods and achieves state-of-the-art performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: As Schelling had said: “The whole world is thoroughly to caught in reason, but the question is: how did it get caught in the network of reason in the ﬁrst place?”
Revised: As Schelling had said: “The whole world is thoroughly to caught in reason, but the question is: how did it get caught in the network of reason in the ﬁrst place?”

Original: (Kuhn, 1942; Žižek & von Schelling, 1997), people found that learning the causal inferences between the variables is a fundamental problem and has many applications in biology, machine learning, medicine, and economics.
Revised: (Kuhn, 1942; Žižek & von Schelling, 1997), people found that learning the causal inferences between the variables is a fundamental problem and has many applications in biology, machine learning, medicine, and economics.

Original: The problem usually is considered as ﬁnding a directed acyclic graph (DAG) from an observational joint distribution.
Revised: The problem usually is considered as ﬁnding a directed acyclic graph (DAG) from an observational joint distribution.

Original: Unfortunately, learning the DAG structure from the observations is proved to be an
Revised: Unfortunately, learning the DAG structure from the observations is proved to be an

Original: NP-hard problem (Chickering, 1995; Chickering et al., 2004).
Revised: NP-hard problem (Chickering, 1995; Chickering et al., 2004).

Original: The problem is generally formulated as the structural equation model (SEM), where the variable of a child node is a function of its parents with additional noises.
Revised: The problem is generally formulated as the structural equation model (SEM), where the variable of a child node is a function of its parents with additional noises.

Original: Depending on the types of functions (linear or non-linear) and noises (Gaussian, Gumbel, etc.), there are several SEM families, e.g., Spirtes et al.
Revised: Depending on the types of functions (linear or non-linear) and noises (Gaussian, Gumbel, etc.), there are several SEM families, e.g., Spirtes et al.

Original: ; Geiger & Heckerman (1994); Shimizu et al.
Revised: ; Geiger & Heckerman (1994); Shimizu et al.

Original: In general, the graph can be identiﬁed from the joint distribution only up to Markov equivalence classes.
Revised: In general, the graph can be identiﬁed from the joint distribution only up to Markov equivalence classes.

Original: Zhang &
Revised: Zhang &

Original: Hyvarinen (2012); Peters et al.
Revised: Hyvarinen (2012); Peters et al.

Original: ; Peters & Bühlmann (2014);
Revised: ; Peters & Bühlmann (2014);

Original: Gao et al.
Revised: Gao et al.

Original: (2020) propose several
Revised: (2020) propose several

Original: SEM forms that make the graph fully identiﬁable from the observed data.
Revised: SEM forms that make the graph fully identiﬁable from the observed data.

Original: Various algorithms had been proposed to deal with the problem.
Revised: Various algorithms had been proposed to deal with the problem.

Original: Search-based algorithms (Chickering, 2002; Friedman & Koller, 2003; Ramsey et al., 2017; Tsamardinos et al., 2006; Aragam & Zhou, 2015; Teyssier & Koller, 2005; Ye et al., 2019; Lv et al., 2021) generally adopt a score (e.g., BIC (Peters et al., 2014) score, Cholesky score (Ye et al., 2019), remove-ﬁll score (Squires et al., 2020)) to measure the ﬁtness of different graphs over data and then search over the legal DAG space to ﬁnd the structure that achieves the highest score.
Revised: Search-based algorithms (Chickering, 2002; Friedman & Koller, 2003; Ramsey et al., 2017; Tsamardinos et al., 2006; Aragam & Zhou, 2015; Teyssier & Koller, 2005; Ye et al., 2019; Lv et al., 2021) generally adopt a score (e.g., BIC (Peters et al., 2014) score, Cholesky score (Ye et al., 2019), remove-ﬁll score (Squires et al., 2020)) to measure the ﬁtness of different graphs over data and then search over the legal DAG space to ﬁnd the structure that achieves the highest score.

Original: However, exhaustive search over the legal DAG space is infeasible when p is large (e.g., there are 4 .
Revised: However, exhaustive search over the legal DAG space is infeasible when p is large (e.g., there are 4 .

Original: 1e 18 DAGs for p = 10 (Sloane et al., 2003)).
Revised: 1e 18 DAGs for p = 10 (Sloane et al., 2003)).

Original: Those algorithms go in quest of a trade-off between the performance and the time complexity.
Revised: Those algorithms go in quest of a trade-off between the performance and the time complexity.

Original: Since Zheng et al.
Revised: Since Zheng et al.

Original: (2018) proposed an approach that converts the traditional combinatorial optimization problem into a continuous program, many methods (Yu et al., 2019; Lee et al., 2019; Ng et al., 2019a;b; Zheng et al., 2020; Lachapelle et al., 2020; Squires et al., 2020; Zhu et al., 2021) have been proposed.
Revised: (2018) proposed an approach that converts the traditional combinatorial optimization problem into a continuous program, many methods (Yu et al., 2019; Lee et al., 2019; Ng et al., 2019a;b; Zheng et al., 2020; Lachapelle et al., 2020; Squires et al., 2020; Zhu et al., 2021) have been proposed.

Original: Those algorithms formalize the problem as a data reconstruction task with various differentiable constraints on the DAG adjacent matrix and solve it via the augmented Lagrangian method.
Revised: Those algorithms formalize the problem as a data reconstruction task with various differentiable constraints on the DAG adjacent matrix and solve it via the augmented Lagrangian method.

Original: These algorithms are able to utilize neural networks to approximate the complicated relations between the features in the observed data and achieve good performances.
Revised: These algorithms are able to utilize neural networks to approximate the complicated relations between the features in the observed data and achieve good performances.

Original: Recently, reinforcement learning based algorithms (Zhu et al., 2020; Wang et al., 2021) also improved the performance by exploring the possible DAG structure candidates.
Revised: Recently, reinforcement learning based algorithms (Zhu et al., 2020; Wang et al., 2021) also improved the performance by exploring the possible DAG structure candidates.

Original: The algorithms update the  via policy gradient as long as it explored a better DAG structure with a higher reward which measures how well an explored structure meets the requirement of DAG and the observed data.
Revised: The algorithms update the parameters of the model via policy gradient as long as it explored a better DAG structure with a higher reward which measures how well an explored structure meets the requirement of DAG and the observed data.

Original: Topology order search algorithms (TOSA)
Revised: Topology order search algorithms (TOSA)

Original: (Ghoshal & Honorio, 2017; 2018; Chen et al., 2019; Gao et al., 2020; Park, 2020) decompose the DAG learning problem into two phases: (i) Topology order learning via conditional variance of the observed data; (ii) Graph estimation depends on the learned topology order.
Revised: (Ghoshal & Honorio, 2017; 2018; Chen et al., 2019; Gao et al., 2020; Park, 2020) decompose the DAG learning problem into two phases: (i) Topology order learning via conditional variance of the observed data; (ii) Graph estimation depends on the learned topology order.

Original: Those algorithms reduce the computation complexity into polynomial time and are guaranteed to recover the DAG structure under some identiﬁable assumptions.
Revised: Those algorithms reduce the computation complexity into polynomial time and are guaranteed to recover the DAG structure under some identiﬁable assumptions.

Original: Our method in this paper is also a topology order search algorithm and it merges the two phases in TOSA into one.
Revised: Our method in this paper is also a topology order search algorithm and it merges the two phases in TOSA into one.

Original: In each iteration, it attempts to ﬁnd a child or a contemporary of the current node.
Revised: In each iteration, it attempts to ﬁnd a child or a contemporary of the current node.

Original: Meanwhile, it also determines the corresponding column vector of the adjacent matrix.
Revised: Meanwhile, it also determines the corresponding column vector of the adjacent matrix.

Original: The mergence brings three main differences: First, the topology order in TOSA is recovered purely based on the conditional variance of the observed data, whereas our method may also take the sparsity of the adjacent matrix into account; Second, the graph LASSO methods, which are commonly adopted to estimate the graph in the second phase in TOSA, encourage the sparsity of the precision matrix, whereas our method is able to encourage the sparsity of the adjacent matrix; Third, the time complexity is reduced signiﬁcantly.
Revised: The mergence brings three main differences: First, the topology order in TOSA is recovered purely based on the conditional variance of the observed data, whereas our method may also take the sparsity of the adjacent matrix into account; Second, the graph LASSO methods, which are commonly adopted to estimate the graph in the second phase in TOSA, encourage the sparsity of the precision matrix, whereas our method is able to encourage the sparsity of the adjacent matrix; Third, the time complexity is reduced signiﬁcantly.

Original: To be speciﬁc, the time complexity of our algorithm isO
Revised: To be speciﬁc, the time complexity of our algorithm is

Original:  ( p 2 n + p 3 ) , while the fastest algorithm before is O ( p 5 n ) (Park, 2020; Gao et al., 2020).
Revised: O ( p 2 n + p 3 ) , while the fastest algorithm before is O ( p 5 n ) (Park, 2020; Gao et al., 2020).

Original: Here p and n are the numbers of nodes and samples, respectively.
Revised: Here p and n are the numbers of nodes and samples, respectively.

Original: In addition, under proper assumptions, we show that our algorithm takes only O (log( p/(cid:15) ))
Revised: In addition, under proper assumptions, we show that our algorithm takes only O (log( p/(cid:15) )

Original:  samples to exactly recover the DAG with probability at least 1 − (cid:15) .
Revised: R ) samples to exactly recover the DAG with probability at least 1 − (cid:15) where R is the bound of l 2 norm of the observed random vector.

Original: Compared with previous TOSA algorithms, the sample complexity of our method is much better.
Revised: Compared with previous TOSA algorithms, the sample complexity of our method is much better.

Original: Experimental results on synthetic data sets, proteins data sets, and knowledge base data set demonstrate the efﬁciency and effectiveness of our algorithm.
Revised: Experimental results on synthetic data sets, proteins data sets, and knowledge base data set demonstrate the efﬁciency and effectiveness of our algorithm.

Original: For synthetic data sets, compared with previous baselines, our algorithm improves the performance with a signiﬁcant margin and at least tens or hundreds of times faster.

Original: For the proteins data set, we achieve state-of-the-art performance.

Original: For the knowledge base data set, we can observe many reasonable structures of the discovered DAG.

Original: Our code is uploaded as supplementary material and will be open-sourced upon the acceptance of this paper.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: In Section 2, we present our algorithm together with the theoretical analysis.
Revised: In Section 2, we present our algorithm together with the theoretical analysis.

Original: In Section 3, numerical results on synthetic data sets, proteins data set, and knowledge base data set are given.
Revised: In Section 3, numerical results on synthetic data sets, proteins data set, and knowledge base data set are given.

Original: Finally, the paper is concluded in Section 4.
Revised: Finally, the paper is concluded in Section 4.

Original: Notations.
Revised: Notations.

Original: The symbol (cid:107) · (cid:107) stands for the Euclid norm of a vector or the spectral norm of a matrix.
Revised: The symbol (cid:107) · (cid:107) stands for the Euclid norm of a vector or the spectral norm of a matrix.

Original: For a vector x = [ x 1 , x 2 , . . .
Revised: For a vector x = [ x 1 , x 2 , . . .

Original: , x p ] ∈ R p , (cid:107) · (cid:107) 1 stands for the (cid:96) 1 -norm, i.e., (cid:107) x 1 (cid:107) =
Revised: , x p ] ∈ R p , (cid:107) · (cid:107) 1 stands for the (cid:96) 1 -norm, i.e., (cid:107) x 1 (cid:107) =

Original: (cid:80) pi =

Revised: (cid:80) pi =

Original: | x i | .
Revised: | x i | .

Original: For a matrix X =
Revised: For a matrix X =

Original: [ X ij ] ∈ R m × n , (cid:107) · (cid:107) 2 , ∞ stands for the two-to-inﬁnity norm, i.e., (cid:107) X (cid:107) 2 , ∞ =
Revised: [ X ij ] ∈ R m × n , (cid:107) · (cid:107) 2 , ∞ stands for the two-to-inﬁnity norm, i.e., (cid:107) X (cid:107) 2 , ∞ =

Original: max 1 ≤ i ≤ m (cid:107) X i, : (cid:107) ; (cid:107) · (cid:107) max stands for the max norm, (cid:107) X (cid:107) max = max i,j | X ij | .
Revised: max 1 ≤ i ≤ m (cid:107) X i, : (cid:107) ; (cid:107) · (cid:107) max stands for the max norm, (cid:107) X (cid:107) max = max i,j | X ij | .

Original:  C AUSAL D ISCOVERY VIA C HOLESKY F ACTORIZATION (CDCF)
Revised:  C AUSAL D ISCOVERY VIA C HOLESKY F ACTORIZATION (CDCF)

Original: In this section, we ﬁrst present some preliminaries on DAG, then motivating our algorithm.
Revised: In this section, we ﬁrst present some preliminaries on DAG, then motivating our algorithm.

Original: Next, the detailed algorithm and theoretical guarantees for the exact recovery of the algorithm are given.
Revised: Next, the detailed algorithm and theoretical guarantees for the exact recovery of the algorithm are given

Original:  2. P RELIMINARIES
Revised: . 2. P RELIMINARIES

Original: We assume the observed data is entailed by a DAG G =
Revised: We assume the observed data is entailed by a DAG G =

Original: ( p, V, E ) , where p is the number of nodes, V = { v 1 , ..., v p } and E = { ( v i , v j ) | i, j ∈ { 1 , ...p }} represent the set of nodes and edges, respectively.
Revised: ( p, V, E ) , where p is the number of nodes, V = { v 1 , ..., v p } and E = { ( v i , v j ) | i, j ∈ { 1 , ...p }} represent the set of nodes and edges, respectively.

Original: Each node v i is corresponding to a random variable X i .
Revised: Each node v i is corresponding to a random variable X i .

Original: The observed data matrix X =
Revised: The observed data matrix X =

Original: [ x 1 , ..., x p ] ∈ R n × p where x i is consisting of n i.i.d observations of the random variable X i .
Revised: [ x 1 , ..., x p ] ∈ R n × p where x i is consisting of n i.i.d observations of the random variable X i .

Original: The joint distribution of X is P ( X ) =
Revised: The joint distribution of X is P ( X ) =

Original: (cid:81) pi
Revised: (cid:81) pi

Original: =1 P ( X i | Pa G ( X i )) , where Pa G ( X i ) :=
Revised: =1 P ( X i | Pa G ( X i )) , where Pa G ( X i ) :=

Original: { X j | ( v i , v j ) ∈ E } is the parents of node X i .
Revised: { X j | ( v i , v j ) ∈ E } is the parents of node X i .

Original: Given X , we seek to recover the latent DAG topology structure for the joint probability distribution (Hoyer et al., 2008; Peters et al., 2017).
Revised: Given X , we seek to recover the latent DAG topology structure for the joint probability distribution (Hoyer et al., 2008; Peters et al., 2017).

Original: Generally, X is modeled via a structural equation model (SEM) with the form
Revised: Generally, X is modeled via a structural equation model (SEM) with the form

Original: [Equation]
Revised: [Equation]

Original: where f i is an arbitrary function representing the relation between X i and its parents, N i is the jointly independent noise variable.
Revised: where f i is an arbitrary function representing the relation between X i and its parents, N i is the jointly independent noise variable.

Original: In this paper, we focus on the linear SEM deﬁned by
Revised: In this paper, we focus on the linear SEM deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where w i ∈ R p is a weighted column vector.
Revised: where w i ∈ R p is a weighted column vector.

Original: Let W =
Revised: Let W =

Original: [ w 1 , . . . , w p ] ∈ R p × p be the weighted adjacency matrix, N = [ n 1 , . . . , n p ] ∈ R n × p be an additive independent noise matrix, where n i is n i.i.d observations following the noise variable N i .
Revised: [ w 1 , . . . , w p ] ∈ R p × p be the weighted adjacency matrix, N = [ n 1 , . . . , n p ] ∈ R n × p be an additive independent noise matrix, where n i is n i.i.d observations following the noise variable N i .

Original: Then the linear SEM model can be formulated as
Revised: Then the linear SEM model can be formulated as

Original: [Equation]
Revised: [Equation]

Original: We assume the noise deviation of the child variable is approximately larger than that of its parents (see Theorem 2.1 for details).
Revised: We assume the noise deviation of the child variable is approximately larger than that of its parents (see Theorem 2.1 for details).

Original: Following this assumption, a classical identiﬁable form of SEM is the linear-Gaussian SEM, where all N i are i.i.d.
Revised: Following this assumption, a classical identiﬁable form of SEM is the linear-Gaussian SEM, where all N i are i.i.d.

Original: and homoscedastic (Peters & Bühlmann, 2014).
Revised: and homoscedastic (Peters & Bühlmann, 2014).

Original: 2. A LGORITHM M OTIVATION
Revised: 2. A LGORITHM M OTIVATION

Original: As proposed in McKay et al.
Revised: As proposed in McKay et al.

Original: ; Nicholson (1975), a graph is DAG if and only if the corresponding weighted adjacent matrix W can be decomposed into
Revised: ; Nicholson (1975), a graph is DAG if and only if the corresponding weighted adjacent matrix W can be decomposed into

Original: [Equation]
Revised: [Equation]

Original: where P is a permutation matrix, T is a strict upper triangular matrix, i.e., T ij = 0 for all i ≤ j .
Revised: where P is a permutation matrix, T is a strict upper triangular matrix, i.e., T ij = 0 for all i ≤ j .

Original: We denote the scaled permuted data matrix as (cid:99) X = 1 √ n XP , the scaled permuted noise matrix as (cid:99) N = 1 √ n NP , and the permutation order [ i ∗ 1 , i ∗ 2 . .
Revised: We denote the scaled permuted data matrix as (cid:99) X = 1 √ n XP , the scaled permuted noise matrix as (cid:99) N = 1 √ n NP , and the permutation order [ i ∗ 1 , i ∗ 2 . .

Original: , i ∗ p ] =
Revised: , i ∗ p ] =

Original: . , p ] P .
Revised: . , p ] P .

Original: We can rewrite (1) as (cid:99)
Revised: We can rewrite (1) as (cid:99)

Original: [Equation]
Revised: [Equation]

Original: Then it follows that
Revised: Then it follows that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) Σ 2 ∗ is the covariance matrix of the noise variables, (cid:98) Σ is upper triangular – the Cholesky factor of (cid:98) Σ 2 ∗ .
Revised: where (cid:98) Σ 2 ∗ is the covariance matrix of the noise variables, (cid:98) Σ is upper triangular – the Cholesky factor of (cid:98) Σ 2 ∗ .

Original: Let the diagonal entries of (cid:98) Σ be σ 2 i ∗ 1 , σ 2 i ∗ 2 , . . .
Revised: Let the diagonal entries of (cid:98) Σ be σ 2 i ∗ 1 , σ 2 i ∗ 2 , . . .

Original: , σ 2 i ∗ p .
Revised: , σ 2 i ∗ p .

Original: We know that σ 2 i ∗ k is the conditional variance of N i ∗ k .
Revised: We know that σ 2 i ∗ k is the conditional variance of N i ∗ k .

Original: Now using (3) and (4), we have the covariance matrix of the permuted data:
Revised: Now using (3) and (4), we have the covariance matrix of the permuted data:

Original: [Equation]
Revised: [Equation]

Original: Let L
Revised: Let L

Original: = ( I − T ) − T (cid:98) Σ T , then (cid:98) C ∗ = LL T , which is the Cholesky factorization of the covariance matrix (cid:98) C ∗ since L is lower triangular.
Revised: = ( I − T ) − T (cid:98) Σ T , then (cid:98) C ∗ = LL T , which is the Cholesky factorization of the covariance matrix (cid:98) C ∗ since L is lower triangular.

Original: Furthermore, we can see that the diagonal entries of L are the same as that of (cid:98) Σ , i.e., L kk =
Revised: Furthermore, we can see that the diagonal entries of L are the same as that of (cid:98) Σ , i.e., L kk =

Original: σ i ∗ k , the conditional variances of X i ∗ k and N i ∗ k are the same.
Revised: σ i ∗ k , the conditional variances of X i ∗ k and N i ∗ k are the same.

Original: The task becomes to ﬁnd the permutation i ∗ =
Revised: The task becomes to ﬁnd the permutation i ∗ =

Original: [ i ∗ 1 , i ∗ 2 , . .
Revised: [ i ∗ 1 , i ∗ 2 , . .

Original: , i ∗ p ] and an upper triangular matrix U such that U − T U − 1 is a good approximation of the empirical estimation of the permuted covariance matrix (cid:98) C = 1 n X T: , i ∗ X : , i ∗ , and U satisﬁes some additional constraints, such as the sparsity, etc.
Revised: , i ∗ p ] and an upper triangular matrix U such that U − T U − 1 is a good approximation of the empirical estimation of the permuted covariance matrix (cid:98) C = 1 n X T: , i ∗ X : , i ∗ , and U satisﬁes some additional constraints, such as the sparsity, etc.

Original: 2. A LGORITHM
Revised: 2. A LGORITHM

Original: We iteratively ﬁnd the permutation i and calculate U via the Cholesky factorization.
Revised: We iteratively ﬁnd the permutation i and calculate U via the Cholesky factorization.

Original: Assume that i k − 1 =
Revised: Assume that i k − 1 =

Original: [ i 1 , . .
Revised: [ i 1 , . .

Original: i k − 1 ] and U k − 

Revised: i k − 1 ] and U k −

Original: = U 1: k − 1 , 1: k − 1 are settled, and we have
Revised: = U 1: k − 1 , 1: k − 1 are settled, and we have

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 is a diagonal augmentation parameter which we will give detailed discussion latter.
Revised: where λ > 0 is a diagonal augmentation parameter which we will give detailed discussion latter.

Original: Next, we show how to ﬁnd i k
Revised: Next, we show how to ﬁnd i k

Original: and the last column of U k
Revised: and the last column of U k

Original: [Equation]
Revised: [Equation]

Original: For the time being
Revised: For the time being

Original: [Equation]
Revised: [Equation]

Original: let us assume i k is known
Revised: let us assume i k is known

Original: [Equation]
Revised: [Equation]

Original: we show how to compute the last column of U k .
Revised: we show how to compute the last column of U k .

Original: Let
Revised: Let

Original: U − 1 k
Revised: U − 1 k

Original: [Equation]
Revised: [Equation]

Original: , then
Revised: , then

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Set U k
Revised: Set U k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:21) ; 14: end for 15: return A =
Revised: (cid:21) ; 14: end for 15: return A =

Original: U
Revised: U

Original: , ω ))] REVERSE i ) , REVERSE ( i ) .
Revised: , ω ))] REVERSE i ) , REVERSE ( i ) .

Original: where the last equality dues to (6).
Revised: where the last equality dues to (6).

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: And direct calculation gives rise to
Revised: And direct calculation gives rise to

Original: [Equation]
Revised: [Equation]

Original: By (8), once i k is settled, we can obtain the last column of U k .
Revised: By (8), once i k is settled, we can obtain the last column of U k .

Original: Our task remains to select i k from { 1 , . .
Revised: Our task remains to select i k from { 1 , . .

Original: . , p } \ { i 1 , . .
Revised: . , p } \ { i 1 , . .

Original: , i k − 1 } .
Revised: , i k − 1 } .

Original: There are several ways to accomplish this task.
Revised: There are several ways to accomplish this task.

Original: We propose three criteria to select i k .
Revised: We propose three criteria to select i k .

Original: First, we need to compute α j and y j by (7) for all possible j
Revised: First, we need to compute α j and y j by (7) for all possible j

Original: ( i j ∈ { 1 , . . .
Revised: ( i j ∈ { 1 , . . .

Original: , p } \ { i 1 , . . .
Revised: , p } \ { i 1 , . . .

Original: , i k − 1 } ).
Revised: , i k − 1 } ).

Original: Then we select i k according to one of the following criteria:
Revised: Then we select i k according to one of the following criteria:

Original: (V) i
Revised: (V) i

Original: k = arg min k ≤ j ≤ p α 2 j .
Revised: k = arg min k ≤ j ≤ p α 2 j .

Original: Under the assumption that the noise variance of the child variable is approximately larger than that of its parents, it is reasonable/natural to select the index that has the lowest estimation of the noise variance.
Revised: Under the assumption that the noise variance of the child variable is approximately larger than that of its parents, it is reasonable/natural to select the index that has the lowest estimation of the noise variance.

Original: This criterion is guaranteed to ﬁnd the correct permutation i ∗ with high probability, which is shown in Section 2.4.
Revised: This criterion is guaranteed to ﬁnd the correct permutation i ∗ with high probability, which is shown in Section 2.4.

Original: (S)
Revised: (S)

Original: i k = arg
Revised: i k = arg

Original: min k ≤ j ≤ p (cid:107) U k − 1 y j (cid:107) 1 .
Revised: min k ≤ j ≤ p (cid:107) U k − 1 y j (cid:107) 1 .

Original: Using (
Revised: Using (

Original: 3) and (6), we know that U p intends to estimate ( I
Revised: 3) and (6), we know that U p intends to estimate ( I

Original: − T
Revised: − T

Original: ) (cid:98)
Revised: ) (cid:98)

Original: Σ −
Revised: Σ −

Original: When the adjacent matrix T is sparse and the noise variables are independent (i.
Revised: When the adjacent matrix T is sparse and the noise variables are independent (i.

Original: e.
Revised: e.

Original: (cid:98) Σ is diagonal
Revised: (cid:98) Σ is diagonal

Original: ), we would like to select the index that leading to the most sparse column of
Revised: ), we would like to select the index that leading to the most sparse column of

Original: U k .
Revised: U k .

Original: This criterion is especially useful when the number of samples is small, see Tables B.1,
Revised: This criterion is especially useful when the number of samples is small, see Tables B.1,

Original: B
Revised: B

Original: and B.
Revised: and B.

Original: in
Revised: in

Original: appendix
Revised: appendix

Original: (VS) i
Revised: (VS) i

Original: k = arg min k ≤ j ≤ p (cid:107) U k − 1 y j (cid:107) 1 (cid:113)(cid:12)(cid:12)
Revised: k = arg min k ≤ j ≤ p (cid:107) U k − 1 y j (cid:107) 1 (cid:113)(cid:12)(cid:12)

Original: α 2 j − 1 k − 1 (cid:80) k − 1 h
Revised: α 2 j − 1 k − 1 (cid:80) k − 1 h

Original: = 

Revised: =

Original: [ U k − 1 ] 2 hh (cid:12)(cid:12) .
Revised: [ U k − 1 ] 2 hh (cid:12)(cid:12) .

Original: We empirically combine criterion (V
Revised: We empirically combine criterion (V

Original: and
Revised: and

Original: criterion
Revised: criterion

Original: (S
Revised: (S

Original: together to take both
Revised: together to take both

Original: aspects (
Revised: aspects (

Original: variance
Revised: variance

Original: and sparsity)
Revised: and sparsity)

Original: into
Revised: into

Original: account
Revised: account

Original: Numerically, we found that this criterion achieves the best performance in real-world data.
Revised: Numerically, we found that this criterion achieves the best performance in real-world data.

Original: The diagonal augmentation trick in (6) is commonly used for an invertible and good conditioned estimation of the covariance matrix (see e.g., (Ledoit & Wolf, 2004)).
Revised: The diagonal augmentation trick in (6) is commonly used for an invertible and good conditioned estimation of the covariance matrix (see e.g., (Ledoit & Wolf, 2004)).

Original: Such a trick not only ensures that our algorithm does not break down due to the singularity of the sample covariance matrix, but also stabilizes the Cholesky factorization, especially when the sample is insufﬁcient.
Revised: Such a trick not only ensures that our algorithm does not break down due to the singularity of the sample covariance matrix, but also stabilizes the Cholesky factorization, especially when the sample is insufﬁcient.

Original: In addition, by setting λ = O ( log pn ) , the error bound between the population covariance matrix and the augmented sample covariance matrix does not become worse (see Lemma A.2 in the appendix).
Revised: In addition, by setting λ = O ( log pn ) , the error bound between the population covariance matrix and the augmented sample covariance matrix does not become worse (see Lemma A.2 in the appendix).

Original: This trick signiﬁcantly improves the ability to recover the DAG, especially when the samples are insufﬁcient, see Tables B.4, B.5 and B.6 in appendix.
Revised: This trick signiﬁcantly improves the ability to recover the DAG, especially when the samples are insufﬁcient, see Tables B.4, B.5 and B.6 in appendix.

Original: The detailed algorithm is summarized in Algorithm 1.
Revised: The detailed algorithm is summarized in Algorithm 1.

Original: Some comments and implementation details follow.
Revised: Some comments and implementation details follow.

Original: Line 4, we select the very initial value (cid:96) =
Revised: Line 4, we select the very initial value (cid:96) =

Original: arg min {(cid:107) X : ,i 1 (cid:107) , (cid:107) X : ,i 2 (cid:107) , . . .
Revised: arg min {(cid:107) X : ,i 1 (cid:107) , (cid:107) X : ,i 2 (cid:107) , . . .

Original: , (cid:107) X : ,i p (cid:107)} .
Revised: , (cid:107) X : ,i p (cid:107)} .

Original: Line 5, we exchange i 

Revised: Line 5, we exchange i

Original: and i (cid:96) in i and calculate U 1 = (cid:113)
Revised: and i (cid:96) in i and calculate U 1 = (cid:113)

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: Lines 6 to 14, we iteratively calculate
Revised: Lines 6 to 14, we iteratively calculate

Original: U k and update permutation order i until all the indices are settled.
Revised: U k and update permutation order i until all the indices are settled.

Original: Line 15, we truncate U , take its strict upper triangular part (denoted by “ TRIU ”) and re-permute the predicted adjacent matrix back to the original order according to the permutation order i .
Revised: Line 15, we truncate U , take its strict upper triangular part (denoted by “ TRIU ”) and re-permute the predicted adjacent matrix back to the original order according to the permutation order i .

Original: Speciﬁcally, the truncation is done column-wisely.
Revised: Speciﬁcally, the truncation is done column-wisely.

Original: By (8), the value of [ U p ] : ,k is inversely proportional to α k .
Revised: By (8), the value of [ U p ] : ,k is inversely proportional to α k .

Original: So, for column k , we set ω
Revised: So, for column k , we set ω

Original: k = ωα k , and do the truncation: [ U p ]
Revised: k = ωα k , and do the truncation: [ U p ]

Original: ik is set to zero if | [ U p ] ik | < ω k .
Revised: ik is set to zero if | [ U p ] ik | < ω k .

Original: On output, node i connects to node j in G if | A ij | > 0 .
Revised: On output, node i connects to node j in G if | A ij | > 0 .

Original: Time Complexity Note that we do not have to re-calculate the matrix multiplication of
Revised: Time Complexity Note that we do not have to re-calculate the matrix multiplication of

Original: X T: , i k − 1 X : ,i j in line 8 since we can calculate C at the cost of O ( p 2 n ) at ﬁrst.
Revised: X T: , i k − 1 X : ,i j in line 8 since we can calculate C at the cost of O ( p 2 n ) at ﬁrst.

Original: Besides, at step k , we have already calculate U T k − 2 X T: , i k − 1 X : ,i j at previous step, we only need to calculate the last entry of y j , which is the inner product between two k dimensional vectors, at the cost of O ( p ) in worst case.
Revised: Besides, at step k , we have already calculate U T k − 2 X T: , i k − 1 X : ,i j at previous step, we only need to calculate the last entry of y j , which is the inner product between two k dimensional vectors, at the cost of O ( p ) in worst case.

Original: Overall, the time complexity of CDCF is O ( p 3 + p 2 n ) .
Revised: Overall, the time complexity of CDCF is O ( p 3 + p 2 n ) .

Original: When n > p , the complexity becomes
Revised: When n > p , the complexity becomes

Original: O ( p 2 n ) , which is equivalent to the complexity of calculating the covariance matrix.
Revised: O ( p 2 n ) , which is equivalent to the complexity of calculating the covariance matrix.

Original: Additionally, the inner loop (lines 7 to 10) of CDCF can be done in parallel, which makes the algorithm friendly to run on GPU and suitable for large scale calculations.
Revised: Additionally, the inner loop (lines 7 to 10) of CDCF can be done in parallel, which makes the algorithm friendly to run on GPU and suitable for large scale calculations.

Original: 2. E XACT DAG S TRUCTURE R ECOVERY
Revised: 2. E XACT DAG S TRUCTURE R ECOVERY

Original: The following theorem tells that our algorithm is able to recover the DAG exactly with high probability under proper assumptions.
Revised: The following theorem tells that our algorithm is able to recover the DAG exactly with high probability under proper assumptions.

Original: Theorem 2.1 Let x ∈ R p be a zero-mean random vector, C
Revised: Theorem 2.1 Let x ∈ R p be a zero-mean random vector, C

Original: = E ( xx ∗ ) ∈
Revised: = E ( xx ∗ ) ∈

Original: R p × p be the covariance matrix.
Revised: R p × p be the covariance matrix.

Original: Assume the random vector x is bounded: (cid:107) x (cid:107) 2 ≤ R .
Revised: Assume the random vector x is bounded: (cid:107) x (cid:107) 2 ≤ R .

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x n be n independent samples, (cid:98) C = 1 n (cid:80) nk
Revised: , x n be n independent samples, (cid:98) C = 1 n (cid:80) nk

Original: =1 x k x ∗ k be the sample covariance estimator, and (cid:98) C
Revised: =1 x k x ∗ k be the sample covariance estimator, and (cid:98) C

Original: λ
Revised: λ

Original: = (cid:98) C
Revised: = (cid:98) C

Original: + λ I , where λ ≥ 0 .
Revised: + λ I , where λ ≥ 0 .

Original: Let the Cholesky factorizations of C and (cid:98) C λ be C = LL T and (cid:98) C λ = (cid:98) L (cid:98) L T , respectively, where L and (cid:98) L are both lower triangular, λ = γ log p n R and γ ≥ 0 is a parameter.
Revised: Let the Cholesky factorizations of C and (cid:98) C λ be C = LL T and (cid:98) C λ = (cid:98) L (cid:98) L T , respectively, where L and (cid:98) L are both lower triangular, λ = γ log p n R and γ ≥ 0 is a parameter.

Original: For the linear SEM model (1), assume (2) and (4), and for k ∈
Revised: For the linear SEM model (1), assume (2) and (4), and for k ∈

Original: Pa G ( j ) , δ = inf k ∈
Revised: Pa G ( j ) , δ = inf k ∈

Original: Pa G ( j ) δ jk > 0 , where
Revised: Pa G ( j ) δ jk > 0 , where

Original: [Equation]
Revised: [Equation]

Original: If n ≥ 4( ξ + γ )log p R δ , then CDCF - V is able to recover P w.p. ≥ 1 − 2 p 1 − 38 ξ .
Revised: If n ≥ 4( ξ + γ )log p R δ , then CDCF - V is able to recover P w.p. ≥ 1 − 2 p 1 − 38 ξ .

Original: In addition, it holds that
Revised: In addition, it holds that

Original: [Equation]
Revised: [Equation]

Original: where TRIU ( U p ) stands for the strictly upper triangular part of U p , U p is the output of outer loop of
Revised: where TRIU ( U p ) stands for the strictly upper triangular part of U p , U p is the output of outer loop of

Original: Algorithm 1 with criterion (V) .
Revised: Algorithm 1 with criterion (V) .

Original: The proofs of Theorem 2.1 are provided in the Appendix A.
Revised: The proofs of Theorem 2.1 are provided in the Appendix A.

Original: The condition δ > 0 is the same as that in Ghoshal & Honorio (2017; 2018);
Revised: The condition δ > 0 is the same as that in Ghoshal & Honorio (2017; 2018);

Original: Chen et al.
Revised: Chen et al.

Original: ; Gao et al.
Revised: ; Gao et al.

Original: ; Park (2020).
Revised: ; Park (2020).

Original: By the upper bound for (cid:107) TRIU ( U p ) − T (cid:107) max , we know that when T is sparse, we may recover its topology structure by truncating U p .
Revised: By the upper bound for (cid:107) TRIU ( U p ) − T (cid:107) max , we know that when T is sparse, we may recover its topology structure by truncating U p .

Original: This theorem also indicates that our algorithm takes O (log( p/(cid:15) ))
Revised: This theorem also indicates that our algorithm takes O (log( p/(cid:15) )

Original:  samples to recover the structure with the probability at least 1 − (cid:15) .
Revised: R ) samples to recover the structure with the probability at least 1 − (cid:15) .

Original: This sample complexity is better than the sample complexities of previous methods, see Table 2.1 for a detailed comparison.
Revised: This sample complexity is better than the sample complexities of previous methods, see Table 2.1 for a detailed comparison.

Revised: In general cases, R ≤ p M where M = max j ( C jj ) .

Revised: When all the nodes are root, R = p M .

Revised: In full

Original: We can also see that our assumptions are more general except the linear assumption on the function type.
Revised: DAG case, R = 74 M . The sample complexity contains M 2 in algorithms LISTEN, EV, and US. When R < M 2 , our algorithm is better than previous algorithms in sample complexity. For instance, if the graph is a sequence of all the nodes, M 2 = p 4 , R = (cid:80) pj =1 ( j 2 )

Revised: =

Revised: O ( p 3 ) .

Revised: In Figure B.1, we give the comparison of R and M 2 in different graph types including ER and SF at different edge levels.

Revised: The result indicates that R < M 2 in general settings.

Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we apply our algorithm to synthetic data sets, proteins data set and knowledge base data set, respectively, to illustrate the efﬁciency and effectiveness of our algorithm.
Revised: In this section [Table] we apply our algorithm to synthetic data sets, proteins data set and knowledge base data set, respectively, to illustrate the efﬁciency and effectiveness of our algorithm

Original: [Table]
Revised: [Table]

Original: 3. L INEAR SEM
Revised: 3. L INEAR SEM

Original: We evaluate the proposed methods on simulated graphs from two well-known ensembles of random graph types:
Revised: We evaluate the proposed methods on simulated graphs from two well-known ensembles of random graph types:

Original: Erdös–Rényi (ER) (Gilbert, 1959) and Scale-free (SF) (Barabási & Albert, 1999).
Revised: Erdös–Rényi (ER) (Gilbert, 1959) and Scale-free (SF) (Barabási & Albert, 1999).

Original: The average edge number per node is denoted after the graph type.
Revised: The average edge number per node is denoted after the graph type.

Original: For example, ER2 represents two edges per node on average.
Revised: For example, ER2 represents two edges per node on average.

Original: After the graph structure is settled, we assign uniformly random edge weights to obtain a weight matrix W .
Revised: After the graph structure is settled, we assign uniformly random edge weights to obtain a weight matrix W .

Original: We generate the observation data X from the linear SEM with three noise distributions: Gaussian, Gumbel, Exponential.
Revised: We generate the observation data X from the linear SEM with three noise distributions: Gaussian, Gumbel, Exponential.

Original: We chose our baseline methods as NOTEARS (Zheng et al., 2018), DAG-GNN (Yu et al., 2019),
Revised: We chose our baseline methods as NOTEARS (Zheng et al., 2018), DAG-GNN (Yu et al., 2019),

Original: CORL (Wang et al., 2021), NPVAR (Gao et al., 2020), and EQVAR (Chen et al., 2019).
Revised: CORL (Wang et al., 2021), NPVAR (Gao et al., 2020), and EQVAR (Chen et al., 2019).

Original: Other methods such as PC algorithm (Spirtes et al., 2000), LiNGAM (Shimizu et al., 2006), FGS (Ramsey et al., 2017), MMHC (Tsamardinos et al., 2006), L1OBS (Schmidt et al., 2007), CAM (Bühlmann et al., 2013), RL-BIC2 (Zhu et al., 2020), A*LASSO (Xiang & Kim, 2013), LISTEN (Ghoshal &
Revised: Other methods such as PC algorithm (Spirtes et al., 2000), LiNGAM (Shimizu et al., 2006), FGS (Ramsey et al., 2017), MMHC (Tsamardinos et al., 2006), L1OBS (Schmidt et al., 2007), CAM (Bühlmann et al., 2013), RL-BIC2 (Zhu et al., 2020), A*LASSO (Xiang & Kim, 2013), LISTEN (Ghoshal &

Original: Honorio, 2018), US (Park, 2020) perform worse than or approximately equal to the selected baselines, and the results can be found in the corresponding papers.
Revised: Honorio, 2018), US (Park, 2020) perform worse than or approximately equal to the selected baseline EQVAR, and the results can be found in the corresponding papers.

Original: Table 3.1 presents the structural Hamming distance (SHD) of baseline methods and our method on 3000 samples ( n = 3000 ).
Revised: Table 3.1 presents the structural Hamming distance (SHD) of baseline methods and our method on 3000 samples ( n = 3000 ).

Original: Nodes number p is noted in the ﬁrst column.
Revised: Nodes number p is noted in the ﬁrst column.

Original: Graph type and edge level are noted in the second column.
Revised: Graph type and edge level are noted in the second column.

Original: We only report the SHD of different algorithms due to page limitation, and we ﬁnd that other metrics such as true positive rate (TPR), false discovery rate (FDR), false positive rate (FPR), and F1 score have the similar comparative performance with SHD.
Revised: We only report the SHD of different algorithms due to page limitation, and we ﬁnd that other metrics such as true positive rate (TPR), false discovery rate (FDR), false positive rate (FPR), and F1 score have the similar comparative performance with SHD.

Original: We also test bottom-up EQVAR which is equivalent to LISTEN, the result is worse than top-down EQVAR (EV-TD) in this synthesis experiment, so we do not include the result in the table.
Revised: We also test bottom-up EQVAR which is equivalent to LISTEN, the result is worse than top-down EQVAR (EV-TD) in this synthesis experiment, so we do not include the result in the table.

Original: For p =graphs, we only report the result of EV-TD and CDCF since other algorithms spend too much time (longer than a week) to recover a DAG.
Revised: For p =graphs, we only report the result of EV-TD and CDCF since other algorithms spend too much time (longer than a week) to recover a DAG.

Original: We test our algorithms with different variations according to criteria (V, S, VS) introduced in Section 2.3, and with diagonal augmentation trick noted by a “+” as postﬁx.
Revised: We test our algorithms with different variations according to criteria (V, S, VS) introduced in Section 2.3, and with diagonal augmentation trick noted by a “+” as postﬁx.

Original: For example, "CDCF-V" means CDCF with V criterion and λ = 0 , and "CDCF-V+" means CDCF with V criterion and λ
Revised: For example, "CDCF-V" means CDCF with V criterion and λ = 0 , and "CDCF-V+" means CDCF with V criterion and λ

Original: =
Revised: =

Original: O ( log pn ) .
Revised: O ( log pn ) .

Original: The implementation details are in the Appendix B.
Revised: The implementation details are in the Appendix B.

Original: We report the result of CDCF-V+ here, and the results of other CDCF variations can be found in
Revised: We report the result of CDCF-V+ here, and the results of other CDCF variations can be found in

Original: Appendix Table B.4.
Revised: Appendix Table B.4.

Original: We run our methods on ten randomly generated graphs and report the mean and variance in the table.
Revised: We run our methods on ten randomly generated graphs and report the mean and variance in the table.

Original: Figure 3.1 plots the SHD results tested on 100 nodes graph recovering from different sample sizes.
Revised: Figure 3.1 plots the SHD results tested on 100 nodes graph recovering from different sample sizes.

Original:  We choose EV-TD and  high dimension top down (EV-HTD) as baselines when p > n and p ≤ n , respectively.
Revised: In low dimension setting ( p < n ), We choose EV-TD, LISTEN (LTN), and LISTEN with Cholesky estimator (LTN-CH) as baselines. In high dimension setting ( p > n ), we [Figure]

Revised: Graph

Revised: NOTEARS

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: {+EV-TD 0 .0 .0 .
+}
Revised: {+EV-TD 0 .0 .0 .
+}

Original: {+LISTEN 0 .0 .0 .
+}
Revised: {+LISTEN 0 .0 .0 .
+}

Revised: [Table]

Revised: HTD 8 .7 .6 .12 .

Revised: [Table]

Revised: DAG

Revised: [Table]

Revised: NPVAR

Revised: [Table]

Revised: [Table]

Revised: NOTEARS 78 .597 .51 .306 .3237 .

Revised: [Table]

Revised: − − − − choose high dimension top down (EV-HTD) and LISTEN (LTN) as baselines.

Original: We can see from the results, CDCF-V+ achieves signiﬁcantly better performance comparing with previous baselines.
Revised: We can see from the results, CDCF-V+ achieves signiﬁcantly better performance comparing with previous baselines.

Original: Table 3.2 shows the running time which is tested on a 2.3 GHz single Intel Core i5 CPU.
Revised: Table 3.2 shows the running time which is tested on a 2.3 GHz single Intel Core i5 CPU.

Original: Besides, parallel calculation of the matrix multiplication on GPU makes the algorithm even faster.
Revised: Besides, parallel calculation of the matrix multiplication on GPU makes the algorithm even faster.

Original: Recovering 5000 and 10000 nodes graph from 3000 samples on an A100 Nvidia GPU is approximately 400 and 2400 seconds, respectively.
Revised: Recovering 5000 and 10000 nodes graph from 3000 samples on an A100 Nvidia GPU is approximately 400 and 2400 seconds, respectively.

Original: For comparison, EV-TD costs approximately 100 hours to recover a 1000 nodes DAG from 3000 samples.
Revised: For comparison, EV-TD costs approximately 100 hours to recover a 1000 nodes DAG from 3000 samples.

Original: As illustrated in the table, CDCF is approximately dozens or[Figure] [Table] [Table]
Revised: As illustrated in the table, CDCF is approximately dozens or

Original: NOTEARS

Original: [Table]

Original: − − − − hundreds of times faster than EV-TD and LISTEN, and tens of thousands times faster than NOTEARS as CDCF does not have to update the parameters with gradients.

Original: Due to the page limitation, further experiments and discussions of the ablation study (Figures B. to B.14, Tables B.1 to B.6), choice of λ (Tables B.7 to B.10), and performances on different noise distribution (Figures B.1, B.2) and deviation (Tables B.11, B.12, B.13) are given in Appendix B.

Original: [Table]
Revised: [Table]

Original: samples 17 edges
Revised: samples 17 edges

Original: [Table]
Revised: [Table]

Original: NOTEARS
Revised: NOTEARS

Original: [Table]
Revised: [Table]

Original: NOTEARSMLP
Revised: NOTEARSMLP

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: EV-TD
Revised: EV-TD

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: LISTEN 0 .
Revised: LISTEN 0 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: NPVAR
Revised: NPVAR

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DAG-GNN
Revised: DAG-GNN

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 7466 samples 20 edges
Revised: 7466 samples 20 edges

Original: CDCF-
Revised: CDCF-

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: NOTEARS
Revised: NOTEARS

Original: [Table]
Revised: [Table]

Original: NOTEARSMLP
Revised: NOTEARSMLP

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: EV
Revised: EV

Original: [Table]
Revised: [Table]

Original: TD 0 .
Revised: TD 0 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: LISTEN 0 .
Revised: LISTEN 0 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: NPVAR
Revised: NPVAR

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: hundreds of times faster than EV-TD and LISTEN, and tens of thousands times faster than NOTEARS as CDCF does not have to update the parameters with gradients.

Revised: Due to the page limitation, further experiments and discussions of the ablation study (Figures B. to B.15, Tables B.1 to B.6), choice of λ (Tables B.7 to B.10), and performances on different noise distribution (Figures B.2, B.3) and deviation (Tables B.11, B.12, B.13) are given in Appendix B.

Original: 3. P ROTEINS D ATA S ET
Revised: 3. P ROTEINS D ATA S ET

Original: We consider a bioinformatics data set (Sachs et al., 2005) consisting of continuous measurements of expression levels of proteins and phospholipids in the human immune system cells.
Revised: We consider a bioinformatics data set (Sachs et al., 2005) consisting of continuous measurements of expression levels of proteins and phospholipids in the human immune system cells.

Original: This is a widely used data set for research on graphical models, with experimental annotations accepted by the biological research community.
Revised: This is a widely used data set for research on graphical models, with experimental annotations accepted by the biological research community.

Original: Following the previous algorithms setting, we noticed that different previous papers adopted different observations.
Revised: Following the previous algorithms setting, we noticed that different previous papers adopted different observations.

Original: To included them all, we considered the observational 853 samples from the "CD3, CD28" simulation tested by Teyssier & Koller (2005); Lachapelle et al.
Revised: To included them all, we considered the observational 853 samples from the "CD3, CD28" simulation tested by Teyssier & Koller (2005); Lachapelle et al.

Original: (2020); Zhu et al.
Revised: (2020); Zhu et al.

Original: (2020) and all 7466 samples from nine different simulations tested by Zheng et al.
Revised: (2020) and all 7466 samples from nine different simulations tested by Zheng et al.

Original: (2018; 2020); Yu et al.
Revised: (2018; 2020); Yu et al.

Original: We report the experimental results on both settings in Table 3.3.
Revised: We report the experimental results on both settings in Table 3.3.

Original: The implementation codes of the baselines are introduced in the appendix, and we use the default settings of the hyper-parameters provided in their codes.
Revised: The implementation codes of the baselines are introduced in the appendix, and we use the default settings of the hyper-parameters provided in their codes.

Original: The evaluate metric is FDR, TPR, FPR, SHD, predicted nodes number (N), precision (P), F1 score.
Revised: The evaluate metric is FDR, TPR, FPR, SHD, predicted nodes number (N), precision (P), F1 score.

Original: As the recall score is equal to TPR, we do not include it in the table.
Revised: As the recall score is equal to TPR, we do not include it in the table.

Original: In both settings, CDCF-VS+ achieves state-of-the-art performance.
Revised: In both settings, CDCF-VS+ achieves state-of-the-art performance.

Original: 1 Several reasons make the recovered graph not exactly the same as the expected one.
Revised: 1 Several reasons make the recovered graph not exactly the same as the expected one.

Original: The ground truth graph suggested by the paper is mixed with directed and indirect edges.
Revised: The ground truth graph suggested by the paper is mixed with directed and indirect edges.

Original: Under the settings of SEM, the node "PKA" is quite similar to the leaf nodes since most of its edges are indirect while the ground truth graph notes it as the out edges.
Revised: Under the settings of SEM, the node "PKA" is quite similar to the leaf nodes since most of its edges are indirect while the ground truth graph notes it as the out edges.

Original: Non-linear would not be an impact issue here since NOTEARS and our algorithm both achieve decent results.
Revised: Non-linear would not be an impact issue here since NOTEARS and our algorithm both achieve decent results.

Original: In the meantime, we do not deny that further extension of our algorithm to non-linear representation would witness an improvement on this data set.
Revised: In the meantime, we do not deny that further extension of our algorithm to non-linear representation would witness an improvement on this data set.

Original: 3. K NOWLEDGE B ASE D ATA S ET

Original: We test our algorithm on FB15K-237 data set (Toutanova et al., 2015) in which the knowledge is organized as { Subject, Predicate, Object } triplets.

Original: The data set has 15K triplets and 237 types of predicates.

Original: In this experiment, we only consider the single jump predicate between the entities, which have 97 predicates remained.

Original: We want to discover the causal relationships between the predicates.

Original: We organize the observation data as each sample corresponds to an entity with awareness of the position (Subject or Object), and each variable corresponds to a predicate in this knowledge base.

Original: F
Revised: F

Original: P
Revised: P

Original: L
Revised: L

Original: M E T · · · · · · ······
Revised: M E T · · · · · · ······

Original: F
Revised: F

Original: P
Revised: P

Original: L
Revised: L

Original: M
Revised: M

Original: E
Revised: E

Original: T ············ 2.
Revised: T ············ 2.

Original: 2.0 Parent
Revised: 2.0 Parent

Original: Child
Revised: Child

Original: Medicine/Disease
Revised: Medicine/Disease

Original: People/CauseOfDeath Medicine/RiskFactors People/CauseOfDeath
Revised: People/CauseOfDeath Medicine/RiskFactors People/CauseOfDeath

Original: Broadcast/Artist
Revised: Broadcast/Artist

Original: Music/Artist Broadcast/Artist
Revised: Music/Artist Broadcast/Artist

Original: Music/Instrumentalists Tv/
Revised: Music/Instrumentalists Tv/

Original: ProgramCreator
Revised: ProgramCreator

Original: Tv/Languages Tv/ProgramCreator
Revised: Tv/Languages Tv/ProgramCreator

Original: Tv/OriginCountry Tv/Languages
Revised: Tv/OriginCountry Tv/Languages

Original: Tv/OriginCountry Tv/Genre
Revised: Tv/OriginCountry Tv/Genre

Original: Film/Country Film/Director
Revised: Film/Country Film/Director

Original: Media/NetﬂixGenre Film/StoryBy
Revised: Media/NetﬂixGenre Film/StoryBy

Original: Film/Prequel Film/WrittenBy
Revised: Film/Prequel Film/WrittenBy

Original: Film/Genre Disease/RiskFactors Disease/symptom
Revised: Film/Genre Disease/RiskFactors Disease/symptom

Original: Sports/Colors
Revised: Sports/Colors

Original: Sports/Teams [Figure] /Nationality
Revised: Sports/Teams Person/Nationality

Original: [Figure]
Revised: Person/PlaceOfBirth Location/TimeZones Location/Country Education/Campuses Education/SchoolType Olympics/Countries Event/Locations

Original: /PlaceOfBirth [Figure] /Country
Revised: Olympics/Countries People/Language

Original: [Figure]
Revised: We test our algorithm on FB15K-237 data set (Toutanova et al., 2015) in which the knowledge is organized as { Subject, Predicate, Object } triplets.

Original: /Campuses
Revised: The data set has 15K triplets and 237 types of predicates. In this experiment, we only consider the single jump predicate between the entities, which have 97 predicates remained.

Original: [Figure]
Revised: We want to discover the causal relationships between the predicates. We organize the observation data as each sample corresponds to an entity with awareness of the position (Subject or Object), and each variable corresponds to a predicate in this knowledge base.

Original: In Figure 3.2, we give the adjacent weighted matrix of the generated graph and several examples with high conﬁdence (larger than 0.5).
Revised: In Figure 3.2, we give the adjacent weighted matrix of the generated graph and several examples with high conﬁdence (larger than 0.5).

Original: In the left ﬁgure, the label of the axis notes the ﬁrst capital letter of the domain of the relations.
Revised: In the left ﬁgure, the label of the axis notes the ﬁrst capital letter of the domain of the relations.

Original: Some of them are replaced with a dot to save space.
Revised: Some of them are replaced with a dot to save space.

Original: The exact domain nameand the picture with the full predicate name are provided in the appendix.The adjacent matrix plotted here is re-permuted to make the relations in the same domain close to each other. We keep the adjacent matrix inside a domain an upper triangular matrix. Such typology is equivalent to the generated matrix with the original order.
Revised: The exact domain name are provided in the appendix.

Original: The domain clusters are denoted in black boxes at the diagonal of the adjacent matrix.
Revised: The domain clusters are denoted in black boxes at the diagonal of the adjacent matrix.

Original: The red boxes denoted the cross-domain relations that are worth paying attention to.
Revised: The red boxes denoted the cross-domain relations that are worth paying attention to.

Original: Consistent with the innateness of human sense, the recovered relationships inside a domain are denser than those across domains.
Revised: Consistent with the innateness of human sense, the recovered relationships inside a domain are denser than those across domains.

Original: In the cross-domain relations, we found that the predicate in domain "TV" ("T") has many relations with the domain "Film" ("F"), the domain "Broadcast" (last row) have many relations with the domain "Music" ("M").
Revised: In the cross-domain relations, we found that the predicate in domain "TV" ("T") has many relations with the domain "Film" ("F"), the domain "Broadcast" (last row) have many relations with the domain "Music" ("M").

Original: Several cases of the predicted causal relationships are listed on the right side of Figure 3.2, we can see that the discovered indication relations between predicates are quite reasonable.
Revised: Several cases of the predicted causal relationships are listed on the right side of Figure 3.2, we can see that the discovered indication relations between predicates are quite reasonable.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we proposed a topology search algorithm for the DAG structure recovery problem.
Revised: In this paper, we proposed a topology search algorithm for the DAG structure recovery problem.

Original: Our algorithm is better than the existing methods in both time and sample complexities.
Revised: Our algorithm is better than the existing methods in both time and sample complexities.

Original: To be speciﬁc, the time complexity of our algorithm is O
Revised: To be speciﬁc, the time complexity of our algorithm is O

Original: ( p 2 n + p 3 ) , while the fastest algorithm before is O ( p 5 n ) (Park, 2020; Gao et al., 2020), where p and n are the numbers of nodes and samples, respectively.
Revised: ( p 2 n + p 3 ) , while the fastest algorithm before is O ( p 5 n ) (Park, 2020; Gao et al., 2020), where p and n are the numbers of nodes and samples, respectively.

Original: The sample complexity of our algorithm is O (log( p/(cid:15) )) , which is better than others, see Table 2.1 for details.
Revised: The sample complexity of our algorithm is O (log( p/(cid:15) ) R ) where R is the bound of l 2 norm of the observed random vector.

Original: Experimental results on synthetic data sets, proteins data sets, and knowledge base data set demonstrate the efﬁciency and effectiveness of our algorithm.
Revised: Experimental results on synthetic data sets, proteins data sets, and knowledge base data set demonstrate the efﬁciency and effectiveness of our algorithm.

Original: For synthetic data sets, compared with previous baselines, our algorithm improves the performance with a signiﬁcant margin and at least tens or hundreds of times faster.

Original: For the proteins data set, we achieve state-of-the-art performance.

Original: For the knowledge base data set, we can observe many reasonable structures of the discovered DAG.

Original: The proposed algorithm is under the assumption of linear SEM.
Revised: The proposed algorithm is under the assumption of linear SEM.

Original: Generalization of CDCF to nonlinear SEM would be a valuable and important research topic.
Revised: Generalization of CDCF to non-linear SEM would be a valuable and important research topic.

Original: Learning the representation of the observed data for better structure reconstruction via the CDCF algorithm, which requires the algorithm differentiable, is also an attractive problem.
Revised: Learning the representation of the observed data for better structure reconstruction via the CDCF algorithm, which requires the algorithm differentiable, is also an attractive problem.

Original: To deal with the extremely large-scale problems, such as millions of nodes, implementing CDCF via sparse matrix storage and calculation on the GPU is a promising way to further improve computational performance.
Revised: To deal with the extremely large-scale problems, such as millions of nodes, implementing CDCF via sparse matrix storage and calculation on the GPU is a promising way to further improve computational performance.

Original: A P ROOF OF T HEOREM 2. 
Revised: A P ROOF OF T HEOREM 2.

Original: In this section, we ﬁrst give several lemmas, then prove Theorem 2.1.
Revised: In this section, we ﬁrst give several lemmas, then prove Theorem 2.1.

Original: Lemma A.1 (Matrix Bernstein: Bounded Case)
Revised: Lemma A.1 (Matrix Bernstein: Bounded Case)

Original: (Tropp, 2012, Theorem 6.1) Consider a ﬁnite { X k } of independent, random, self-adjoint matrices with dimension d .
Revised: (Tropp, 2012, Theorem 6.1) Consider a ﬁnite { X k } of independent, random, self-adjoint matrices with dimension d .

Original: Assume that E X k = 

Revised: Assume that E X k =

Original: and (cid:107) X k (cid:107) ≤ R almost surely.
Revised: and (cid:107) X k (cid:107) ≤ R almost surely.

Original: Compute the norm of the total variance σ 2 = (cid:107) (cid:80) k
Revised: Compute the norm of the total variance σ 2 = (cid:107) (cid:80) k

Original: E X 2 k (cid:107) .
Revised: E X 2 k (cid:107) .

Original: Then the following chain of inequalities holds for all t ≥ 0 .
Revised: Then the following chain of inequalities holds for all t ≥ 0 .

Original: [Equation]
Revised: [Equation]

Original: The function h ( u ) =
Revised: The function h ( u ) =

Original: (1 + u ) log(1 + u ) − u for h ≥ 0 .
Revised: (1 + u ) log(1 + u ) − u for h ≥ 0 .

Original: Lemma A.2 Let x ∈ R p be a zero-mean random vector,
Revised: Lemma A.2 Let x ∈ R p be a zero-mean random vector,

Original: C = E ( xx ∗ )
Revised: C = E ( xx ∗ )

Original: ∈ R p
Revised: ∈ R p

Original: ×
Revised: ×

Original: p be the covariance matrix.
Revised: p be the covariance matrix.

Original: Assume the random vector x is bounded: (cid:107) x (cid:107) 2 ≤ R .
Revised: Assume the random vector x is bounded: (cid:107) x (cid:107) 2 ≤ R .

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x n be n independent samples, (cid:98) C = 1 n (cid:80) nk
Revised: , x n be n independent samples, (cid:98) C = 1 n (cid:80) nk

Original: =1 x k x ∗ k be the sample covariance estimator, and (cid:98) C
Revised: =1 x k x ∗ k be the sample covariance estimator, and (cid:98) C

Original: λ
Revised: λ

Original: = (cid:98) C
Revised: = (cid:98) C

Original: + λ I , where λ ≥ 0 .
Revised: + λ I , where λ ≥ 0 .

Original: Then w.p. ≥ 1 − p 1 − 38 ξ , it holds that
Revised: Then w.p. ≥ 1 − p 1 − 38 ξ , it holds that

Original: [Equation]
Revised: [Equation]

Original: In addition, w.p. ≥ 1 − p − 38 ξ , it holds that
Revised: In addition, w.p. ≥ 1 − p − 38 ξ , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let S k = 1 n ( x k x ∗ k − C ) for k = 1 , 2 , . . .
Revised: Let S k = 1 n ( x k x ∗ k − C ) for k = 1 , 2 , . . .

Original: , n , then Z = (cid:80) nk =1 S k = (cid:98) C − C , and E S k = 0 ,
Revised: , n , then Z = (cid:80) nk =1 S k = (cid:98) C − C , and E S k = 0 ,

Original: E Z = (cid:80) nk =1 E S k
Revised: E Z = (cid:80) nk =1 E S k

Original: = 0 .
Revised: = 0 .

Original: Note that (cid:107) x k x ∗ k (cid:107) =
Revised: Note that (cid:107) x k x ∗ k (cid:107) =

Original: (cid:107) x k (cid:107) 2 ≤ R , (cid:107) C (cid:107) =
Revised: (cid:107) x k (cid:107) 2 ≤ R , (cid:107) C (cid:107) =

Original: (cid:107) E ( xx ∗ ) (cid:107) ≤ E (cid:107) xx ∗ (cid:107) =
Revised: (cid:107) E ( xx ∗ ) (cid:107) ≤ E (cid:107) xx ∗ (cid:107) =

Original: E (cid:107) x (cid:107) 2 ≤ R , and − C ≤ x k x ∗ k − C ≤ x k x ∗ k , 2 we have
Revised: E (cid:107) x (cid:107) 2 ≤ R , and − C ≤ x k x ∗ k − C ≤ x k x ∗ k , 2 we have

Original: [Equation]
Revised: [Equation]

Original: By calculations, we also have
Revised: By calculations, we also have

Original: [Equation]
Revised: [Equation]

Original: For t = ξR log p n with ξ > 0 , it holds σ 2 R ≤ Rn < ξR log p n = t , and p exp( − 3 t 8 R/n ) = p exp( − 38 ξ log p ) =
Revised: For t = ξR log p n with ξ > 0 , it holds σ 2 R ≤ Rn < ξR log p n = t , and p exp( − 3 t 8 R/n ) = p exp( − 38 ξ log p ) =

Original: p 1 − 38 ξ .
Revised: p 1 − 38 ξ .

Original: Then w.p. ≥ 1 − p 1 − 38 ξ , it holds that (cid:107) C − (cid:98) C (cid:107) ≤ ξ log p n R , and (9) follows immediately.
Revised: Then w.p. ≥ 1 − p 1 − 38 ξ , it holds that (cid:107) C − (cid:98) C (cid:107) ≤ ξ log p n R , and (9) follows immediately.

Original: Similarly, for any 1 ≤ i ≤ p , let s k = 1 n ([ x k ] 

Revised: Similarly, for any 1 ≤ i ≤ p , let s k = 1 n ([ x k ]

Original: i − C ii ) for k = 1 , 2 , . . .
Revised: i − C ii ) for k = 1 , 2 , . . .

Original: , n , then z = (cid:80) nk
Revised: , n , then z = (cid:80) nk

Original: =1 s k = (cid:98) C ii − C ii , and E s k = 0 , E z = 0 .
Revised: =1 s k = (cid:98) C ii − C ii , and E s k = 0 , E z = 0 .

Original: By simple calculations, we have
Revised: By simple calculations, we have

Original: [Equation]
Revised: [Equation]

Original: Let t = ξR log p n with ξ > 0 , then σ 2 i R ≤
Revised: Let t = ξR log p n with ξ > 0 , then σ 2 i R ≤

Original: Rn < t , exp( − 3 t 8 R/n ) =
Revised: Rn < t , exp( − 3 t 8 R/n ) =

Original: exp( − 38 ξ log p ) =
Revised: exp( − 38 ξ log p ) =

Original: p − 38 ξ .
Revised: p − 38 ξ .

Original: Then w.p.
Revised: Then w.p.

Original: ≥ 1 − p − 38 ξ , it holds that (cid:107) C ii − (cid:98) C ii (cid:107) ≤ ξ log p n R , and (10) follows immediately.
Revised: ≥ 1 − p − 38 ξ , it holds that (cid:107) C ii − (cid:98) C ii (cid:107) ≤ ξ log p n R , and (10) follows immediately.

Original: (cid:3)
Revised: (cid:3)

Original: Lemma A.3 Follow the notations and assumptions in Lemma A.2.
Revised: Lemma A.3 Follow the notations and assumptions in Lemma A.2.

Original: Let the Cholesky factorizations of
Revised: Let the Cholesky factorizations of

Original: C and (cid:98) C λ be C = LL T and (cid:98) C λ
Revised: C and (cid:98) C λ be C = LL T and (cid:98) C λ

Original: = (cid:98) L (cid:98) L T , respectively, where L and (cid:98) L are both lower triangular,
Revised: = (cid:98) L (cid:98) L T , respectively, where L and (cid:98) L are both lower triangular,

Original: λ = γ log p n R and γ ≥ 0 is a parameter.
Revised: λ = γ log p n R and γ ≥ 0 is a parameter.

Original: It holds
Revised: It holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, for all i = 1 , 2 , . .
Revised: First, for all i = 1 , 2 , . .

Original: . , p , we notice that |(cid:107)
Revised: . , p , we notice that |(cid:107)

Original: L i, : (cid:107) 2 − (cid:107) (cid:98) L i, : (cid:107) 

Revised: L i, : (cid:107) 2 − (cid:107) (cid:98) L i, : (cid:107)

Original: | = | C ii − [ (cid:98) C λ ] ii | .
Revised: | = | C ii − [ (cid:98) C λ ] ii | .

Original: Then (11) follows from Lemma A.2 immediately since R ≤ R .
Revised: Then (11) follows from Lemma A.2 immediately since R ≤ R .

Original: Next, we show (12).
Revised: Next, we show (12).

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: We know that
Revised: We know that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then it follows from (14) that for i > j
Revised: Then it follows from (14) that for i > j

Original: [Equation]
Revised: [Equation]

Original: First, we give an upper bound for (cid:107) ( I + F ) − 1 (cid:107) .
Revised: First, we give an upper bound for (cid:107) ( I + F ) − 1 (cid:107) .

Original: Using (13)
Revised: Using (13)

Original: [Equation]
Revised: [Equation]

Original: we have ( I + F ) − T ( I + F ) − 

Revised: we have ( I + F ) − T ( I + F ) −

Original: =
Revised: =

Original: ( I + E ) − 1 .
Revised: ( I + E ) − 1 .

Original: It follows
Revised: It follows

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses (15).
Revised: where the last inequality uses (15).

Original: Second, we give upper bound for (cid:107) F i, 1: i − 1 (cid:107) .
Revised: Second, we give upper bound for (cid:107) F i, 1: i − 1 (cid:107) .

Original: It follows from the second equality of (13) that
Revised: It follows from the second equality of (13) that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: (19) where (a) uses (13), (b) uses (12) and (15), (c) uses Lemma A.2.
Revised: (19) where (a) uses (13), (b) uses (12) and (15), (c) uses Lemma A.2.

Original: Substituting (19) and (17) into (16), we get
Revised: Substituting (19) and (17) into (16), we get

Original: [Equation]
Revised: [Equation]

Original: The conclusion follows .
Revised: The conclusion follows .

Original: [Equation]
Revised: [Equation]

Original: Theorem A.4 Let x ∈ R p be a zero-mean random vector, C
Revised: Theorem A.4 Let x ∈ R p be a zero-mean random vector, C

Original: = E ( xx ∗ ) ∈
Revised: = E ( xx ∗ ) ∈

Original: R p × p be the covariance matrix.
Revised: R p × p be the covariance matrix.

Original: Assume the random vector x is bounded: (cid:107) x (cid:107) 2 ≤ R .
Revised: Assume the random vector x is bounded: (cid:107) x (cid:107) 2 ≤ R .

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x n be n independent samples, (cid:98) C = 1 n (cid:80) nk
Revised: , x n be n independent samples, (cid:98) C = 1 n (cid:80) nk

Original: =1 x k x ∗ k be the sample covariance estimator, and (cid:98) C
Revised: =1 x k x ∗ k be the sample covariance estimator, and (cid:98) C

Original: λ
Revised: λ

Original: = (cid:98) C
Revised: = (cid:98) C

Original: + λ I , where λ ≥ 0 .
Revised: + λ I , where λ ≥ 0 .

Original: Let the Cholesky factorizations of C and (cid:98) C λ be C = LL T and (cid:98) C λ = (cid:98) L (cid:98) L T , respectively, where L and (cid:98) L are both lower triangular, λ = γ log p n R and γ ≥ 0 is a parameter.
Revised: Let the Cholesky factorizations of C and (cid:98) C λ be C = LL T and (cid:98) C λ = (cid:98) L (cid:98) L T , respectively, where L and (cid:98) L are both lower triangular, λ = γ log p n R and γ ≥ 0 is a parameter.

Original: For the linear SEM model (1), assume (2) and (4), and for k ∈
Revised: For the linear SEM model (1), assume (2) and (4), and for k ∈

Original: Pa G ( j ) , δ = inf k ∈
Revised: Pa G ( j ) , δ = inf k ∈

Original: Pa G ( j ) δ jk > 0 , where
Revised: Pa G ( j ) δ jk > 0 , where

Original: [Equation]
Revised: [Equation]

Original: If n ≥ 4(
Revised: If n ≥ 4(

Original: [Equation]
Revised: [Equation]

Original: + γ )log pR δ , then CDCF - V is able to recover P w.p. ≥ 1 − 2 p 1 − 38 ξ .
Revised: + γ )log pR δ , then CDCF - V is able to recover P w.p. ≥ 1 − 2 p 1 − 38 ξ .

Original: In addition, it holds that (cid:107)
Revised: In addition, it holds that (cid:107)

Original: [Equation]
Revised: [Equation]

Original: where TRIU ( U p ) stands for the strictly upper triangular part of U p , U p is the output of outer loop of
Revised: where TRIU ( U p ) stands for the strictly upper triangular part of U p , U p is the output of outer loop of

Original: Algorithm 1 with criterion (V) .
Revised: Algorithm 1 with criterion (V) .

Original: Proof.
Revised: Proof.

Original: For SEM model (1), denote (cid:98) C ∗ =
Revised: For SEM model (1), denote (cid:98) C ∗ =

Original: E ( 1 n (cid:99) X T
Revised: E ( 1 n (cid:99) X T

Original: [Equation]
Revised: [Equation]

Original: X ) , (cid:98) Σ
Revised: X ) , (cid:98) Σ

Original: [Equation]
Revised: [Equation]

Original: = E (
Revised: = E (

Original: [Equation]
Revised: [Equation]

Original: n (cid:99) N
Revised: n (cid:99) N

Original: [Equation]
Revised: [Equation]

Original: (cid:99) N )
Revised: (cid:99) N )

Original: = (cid:98) Σ T n (cid:98) Σ n , we have (5), i.e.,
Revised: = (cid:98) Σ T n (cid:98) Σ n , we have (5), i.e.,

Original: [Equation]
Revised: [Equation]

Original: When the permutation i ∗ =
Revised: When the permutation i ∗ =

Original: [ i ∗ 1 , . .
Revised: [ i ∗ 1 , . .

Original: , i ∗ p ] is exactly recovered, then U p in CDCF - V satisﬁes
Revised: , i ∗ p ] is exactly recovered, then U p in CDCF - V satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Denote i ∗ j =
Revised: Denote i ∗ j =

Original: [ i ∗ 1 , . .
Revised: [ i ∗ 1 , . .

Original: , i ∗ j ] for all j = 1 , . . .
Revised: , i ∗ j ] for all j = 1 , . . .

Original: , p .
Revised: , p .

Original: Consider the k th diagonal entries of (22) and (23).
Revised: Consider the k th diagonal entries of (22) and (23).

Original: By calculations, we get
Revised: By calculations, we get

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By Lemma A.2, we have
Revised: By Lemma A.2, we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma A.3, we have w.h.p.
Revised: By Lemma A.3, we have w.h.p.

Original: [Equation]
Revised: [Equation]

Original: Using (24), (25), (27) and (28), we get
Revised: Using (24), (25), (27) and (28), we get

Original: [Equation]
Revised: [Equation]

Original: Assume that i ∗ 1 , . .
Revised: Assume that i ∗ 1 , . .

Original: , i ∗ k − 1 ( k ≥ 1 ) are all correctly recovered.
Revised: , i ∗ k − 1 ( k ≥ 1 ) are all correctly recovered.

Original: And without loss of generality, for k ∈ Pa G ( j ) , we also assume T k : j − 1 ,j (cid:54) = 

Revised: And without loss of generality, for k ∈ Pa G ( j ) , we also assume T k : j − 1 ,j (cid:54) =

Original: (otherwise, j th and k th columns are exchangeable, and i forms another equivalence topology order to the same DAG (Sedgewick & Wayne, 2011)).
Revised: (otherwise, j th and k th columns are exchangeable, and i forms another equivalence topology order to the same DAG (Sedgewick & Wayne, 2011)).

Original: Then we have for k ∈
Revised: Then we have for k ∈

Original: Pa G ( j ) that
Revised: Pa G ( j ) that

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: w.p. ≥ (1 − p − 38 ξ ) 2 .
Revised: w.p. ≥ (1 − p − 38 ξ ) 2 .

Original: Here (a) uses (25), (b) and (g) uses Lemma A.2 and Lemma A.3, (c) uses (24), (d) dues to the assumption σ i ∗ j ≥ σ i ∗ k for k ∈
Revised: Here (a) uses (25), (b) and (g) uses Lemma A.2 and Lemma A.3, (c) uses (24), (d) dues to the assumption σ i ∗ j ≥ σ i ∗ k for k ∈

Original: Pa G ( j ) , (e) uses (24), (g) uses (25).
Revised: Pa G ( j ) , (e) uses (24), (g) uses (25).

Original: Therefore, using n ≥ 4 ξ log pR δ , we have w.p. ≥ (1 − p − 38 ξ ) 2 that
Revised: Therefore, using n ≥ 4 ξ log pR δ , we have w.p. ≥ (1 − p − 38 ξ ) 2 that

Original: [Equation]
Revised: [Equation]

Original: which implies that i ∗ k can be correctly recovered.
Revised: which implies that i ∗ k can be correctly recovered.

Original: So, overall speaking, CDCF - V is able to recover the permutation P w.p. ≥ 1 − 2 p 1 − 38 ξ .
Revised: So, overall speaking, CDCF - V is able to recover the permutation P w.p. ≥ 1 − 2 p 1 − 38 ξ .

Original: The upper bound for (cid:107) TRIU ( U p ) − T (cid:107) max follows from Lemma A.3.
Revised: The upper bound for (cid:107) TRIU ( U p ) − T (cid:107) max follows from Lemma A.3.

Original: The proof is completed.
Revised: The proof is completed.

Original: [Equation]
Revised: [Equation]

Original: B A DDITIONAL E XPERIMENTS
Revised: B A DDITIONAL E XPERIMENTS

Original: Here we provide implementation details and additional experiment results.
Revised: Here we provide implementation details and additional experiment results.

Original: Figures B.1, B.2 provide the results of Gumbel and Exponential noises, respectively.
Revised: Figures B.2, B.3 provide the results of Gumbel and Exponential noises, respectively.

Original: As we can see from the result, our algorithm still performs better than Eqvar method in different noise types.
Revised: As we can see from the result, our algorithm still performs better than Eqvar method in different noise types.

Original: Tables B.1, B.2 , B.3, B.4, B.5, B.6 give results on 100 nodes over different sample sizes and variances of our CDCF methods.
Revised: Tables B.1, B.2 , B.3, B.4, B.5, B.6 give results on 100 nodes over different sample sizes and variances of our CDCF methods.

Original: As noted in Algorithm 1, we have V, S, VS as different criteria to select the current column, "+" representing the sample covariance matrix augmented with the scalar matrix log
Revised: As noted in Algorithm 1, we have V, S, VS as different criteria to select the current column, "+" representing the sample covariance matrix augmented with the scalar matrix log

Original: pn I .
Revised: pn I .

Original: The truncation threshold on column i is ω
Revised: The truncation threshold on column i is ω

Original: i = 3 .
Revised: i = 3 .

Original: 5 /α i , where α i is the diagonal value of the Cholesky factor.
Revised: 5 /α i , where α i is the diagonal value of the Cholesky factor.

Original: According to the results, the algorithm "V+" achieves the best performance as the sample size is relatively large.
Revised: According to the results, the algorithm "V+" achieves the best performance as the sample size is relatively large.

Original: When the sample size is small, the criterion according to sparsity shows very effective performance improvement.
Revised: When the sample size is small, the criterion according to sparsity shows very effective performance improvement.

Original: We also test different choices over λ = β log pn , β ∈ { 0 . 0 , 1 . 

Revised: We also test different choices over λ = β log pn , β ∈ { 0 . 0 , 1 .

Original: ..., 9 . 0 } , the result is given in Table B.7, B.8, B.9, B.10.
Revised: ..., 9 . 0 } , the result is given in Table B.7, B.8, B.9, B.10.

Original: Empirically, β ∈ { 1 . 0 , 2 . 0 } achieves better results.
Revised: Empirically, β ∈ { 1 . 0 , 2 . 0 } achieves better results.

Original: In practice, one can sample a relatively small and labeled sub-graph of the DAG to test the hyper-parameter setting then apply to large unlabeled the DAG graph.
Revised: In practice, one can sample a relatively small and labeled sub-graph of the DAG to test the hyper-parameter setting then apply to large unlabeled the DAG graph.

Original: To test the performance limitation of our methods, we provide the results of SHD on different sample number and node number in Figures B.3 to B.14 where the x-axis represents the sample number (in thousand), the y-axis denotes the node number, the color represents the value of log 2 ( SHD + 1) (the brighter the better).
Revised: To test the performance limitation of our methods, we provide the results of SHD on different sample number and node number in Figures B.4 to B.15 where the x-axis represents the sample number (in thousand), the y-axis denotes the node number, the color represents the value of log 2 ( SHD + 1) (the brighter the better).

Original: We provide the ﬁgures for CDCF-V+, CDCF-S+, and CDCF-VS+ on variances graph and noise types.
Revised: We provide the ﬁgures for CDCF-V+, CDCF-S+, and CDCF-VS+ on variances graph and noise types.

Original: The ﬁgures are drawn on the mean results over ten random seeds.
Revised: The ﬁgures are drawn on the mean results over ten random seeds.

Original: The ﬁgures show that the graph can be exactly recovered on 800 nodes at approximately 6000 samples.
Revised: The ﬁgures show that the graph can be exactly recovered on 800 nodes at approximately 6000 samples.

Original: Comparing
Revised: Comparing

Original: CDCF-V+ with CDCF-S+, we ﬁnd that criterion ( S ) damages the performance when the sample number is relatively large.
Revised: CDCF-V+ with CDCF-S+, we ﬁnd that criterion ( S ) damages the performance when the sample number is relatively large.

Original: When sample number ∈ { 1500 , 3000 } and node number ∈ { 400 , 800 } ,
Revised: When sample number ∈ { 1500 , 3000 } and node number ∈ { 400 , 800 } ,

Original: CDCF-S+ achieves better performance.
Revised: CDCF-S+ achieves better performance.

Original: Such trend can also be demonstrated in Tables B.1, B.2, B.3.
Revised: Such trend can also be demonstrated in Tables B.1, B.2, B.3.

Original: CDCF-VS+ alleviates the poor performance of CDCF-S when the data is sufﬁcient and achieves good performance on real-world data set.
Revised: CDCF-VS+ alleviates the poor performance of CDCF-S when the data is sufﬁcient and achieves good performance on real-world data set.

Original: We also test the performance on linear SEM with monotonously increased noise variance.
Revised: We also test the performance on linear SEM with monotonously increased noise variance.

Original: Concretely, assume the topology order is i
Revised: Concretely, assume the topology order is i

Original: = { i 1 , ..., i p } , we set the noise variance of node k as σ k = 

Revised: = { i 1 , ..., i p } , we set the noise variance of node k as σ k =

Original: + i k /p .
Revised: + i k /p .

Original: We test the results on Gaussian, Gumbel, and Exponential noise with monotonous noise variance.
Revised: We test the results on Gaussian, Gumbel, and Exponential noise with monotonous noise variance.

Original: The results are reported in Tables B.11, B.12 and B.13.
Revised: The results are reported in Tables B.11, B.12 and B.13.

Original: As the results indicated, even with different noise levels, our algorithms achieve good performance and are able to exactly recover the DAG structure when the data is sufﬁcient.
Revised: As the results indicated, even with different noise levels, our algorithms achieve good performance and are able to exactly recover the DAG structure when the data is sufﬁcient.

Original: In the result for knowledge base data set, the axis labels of Figure 3.2 are ‘Film’, ‘People’, ‘Location’, ‘Music’, ‘Education’, ‘Tv’, ‘Medicine’, ‘Sports’, ‘Olympics’, ‘Award’, ‘Time’, ‘Organization’, ‘Language’, ‘MediaCommon’, ‘Inﬂuence’, ‘Dataworld’, ‘Business’, ‘Broadcast’ from left to right for x-axis and top to bottom for y-axis, respectively.
Revised: In the result for knowledge base data set, the axis labels of Figure 3.2 are ‘Film’, ‘People’, ‘Location’, ‘Music’, ‘Education’, ‘Tv’, ‘Medicine’, ‘Sports’, ‘Olympics’, ‘Award’, ‘Time’, ‘Organization’, ‘Language’, ‘MediaCommon’, ‘Inﬂuence’, ‘Dataworld’, ‘Business’, ‘Broadcast’ from left to right for x-axis and top to bottom for y-axis, respectively.

Revised: The adjacent matrix plotted in Figure ?? is re-permuted to make the relations in the same domain close to each other.

Revised: We keep the adjacent matrix inside a domain an upper triangular matrix.

Revised: Such typology is equivalent to the generated matrix with the original order.

Original: Baseline Implementations
Revised: Baseline Implementations

Original: The baselines are implemented via the codes provided from the following links:
Revised: The baselines are implemented via the codes provided from the following links:

Original: • NOTEARS, NOTEARS-MLP: https://github.com/xunzheng/notears • NPVAR:
Revised: • NOTEARS, NOTEARS-MLP: https://github.com/xunzheng/notears • NPVAR:

Original: https://github.com/MingGao97/NPVAR • EQVAR, LISTEN: https://github.com/WY-Chen/EqVarDAG • CORL:
Revised: https://github.com/MingGao97/NPVAR • EQVAR, LISTEN: https://github.com/WY-Chen/EqVarDAG • CORL:

Original: https://github.com/huawei-noah/trustworthyAI/tree/master/gcastle • 
Revised: https://github.com/huawei-noah/trustworthyAI/tree/master/gcastle • DAG-GNN: https://github.com/ﬁshmoon1234/DAG-GNN

Original: [Figure]
Revised: [Figure]

Original: [Figure]ER V+ 174 . S+ 173 .
Revised: [Figure]

Original: VS+ 174 . SF V+ 170 . S+ 170 .
Revised: [Figure]

Original: VS+ 168 . ER V+ 427 . 0 356 . [Table]
Revised: [Table]

Original: [Table][Table] [Figure] N o d e
Revised: [Table]

Original: N u m

Original: [Figure]
Revised: [Table]

Original: r

Original: [Figure] [Figure] [Figure] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: VS 177 . 6 95 . 0 31 .
Revised: VS 177 . 6 95 . 0 31 .

Original: V+ 47 .
Revised: V+ 47 .

Original: S+ 45 .
Revised: S+ 45 .

Original: VS+ 48 .
Revised: VS+ 48 .

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: S+ 54 .

Revised: VS+ 56 .

Original: [Figure]
Revised: [Figure]

Original: S
Revised: CDCF-VS+ 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: VS+ 56 .

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: S 53 .12 .4 .

Revised: S 53 .12 .4 .

Original: VS 59 .13 .3 .1 .0 .
Revised: VS 59 .13 .3 .1 .0 .

Original: V+ 41 .10 .2 .1 .0 .
Revised: V+ 41 .10 .2 .1 .0 .

Original: S+ 37 .9 .3 .

Revised: S+ 37 .9 .3 .

Original: VS+ 

Revised: VS+

Original: [Table]
Revised: [Table]

Original: V 519 .
Revised: V 519 .

Original: S 501 .
Revised: S 501 .

Original: VS 511 .
Revised: VS 511 .

Original: V+ 315 .
Revised: V+ 315 .

Original: S+ 318 .
Revised: S+ 318 .

Original: VS+ 315 .
Revised: VS+ 315 .

Original: [Table] [Table]
Revised: [Table] [Table]