Original: Improving the Efficiency of the PC Algorithm by Using Model-Based Conditional Independence Tests
Revised: Improving the Efficiency of the PC Algorithm by Using Model-Based Conditional Independence Tests

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Learning causal structure is useful in many areas of artificial intelligence, such asplanning, robotics, and explanation.
Revised: Learning causal structure is useful in many areas of artificial intelligence, including planning, robotics, and explanation.

Original: Constraint-basedand hybrid structure learningalgorithms such as PC use conditional independence (CI) tests to learn a causalstructure.
Revised: Constraint-based structure learning algorithms such as PC use conditional independence (CI) tests to infer causal structure.

Original: Traditionally, constraint-based algorithms performthe CI tests witha preference for smaller-sized conditioning sets, partially because the statisticalpower of conventional CI tests declines substantially as the size of the conditioningset increases.
Revised: Traditionally, constraint-based algorithms perform CI tests with a preference for smaller-sized conditioning sets, partially because the statistical power of conventional CI tests declines rapidly as the size of the conditioning set increases.

Original: However, many modern conditional independence tests are modelbased , and these tests use well-regularized models that can perform well even withvery large conditioning sets.
Revised: However, many modern conditional independence tests are model-based , and these tests use well-regularized models that maintain statistical power even with very large conditioning sets.

Original: This suggests an intriguing new strategy for constraintbased algorithms which may result in a reduction of the total number of CI testsperformed: Test variable pairs with large conditioning sets first , as a pre-processingstep that finds some conditional independencies quickly, before moving on tothe more conventional strategy of testing with incrementally larger conditioningsets of sizes (beginning with marginal independence tests).
Revised: This suggests an intriguing new strategy for constraintbased algorithms which may result in a reduction of the total number of CI tests performed: Test variable pairs with large conditioning sets first , as a pre-processing step that finds some conditional independencies quickly, before moving on to the more conventional strategy that favors small conditioning sets.

Original: We propose such apre-processing step for the PC algorithm which relies on performing CI tests on afew randomly selected large conditioning sets.
Revised: We propose such a pre-processing step for the PC algorithm which relies on performing CI tests on a few randomly selected large conditioning sets.

Original: We perform an empirical analysison directed acyclic graphs (DAGs) that correspond to real-world systems and bothan empirical and theoretical analysis for Erd˝os-Renyi DAGs.
Revised: We perform an empirical analysis on directed acyclic graphs (DAGs) that correspond to real-world systems and both empirical and theoretical analyses for Erd ˝ os-Renyi DAGs.

Original: Our results showthat the PC algorithm with our pre-processing step performs far fewer CI teststhan the original PC algorithm, between 0.5%  and 20%, of the CI tests that the PCalgorithm alone performs.
Revised: Our results show that Pre-Processing Plus PC (P3PC) performs far fewer CI tests than the original PC algorithm, between 0.5% to 36% and often less than 10% of the CI tests that the PC algorithm alone performs.

Original: The efficiency gains are particularly significant for the DAGs corresponding to real-world systems.
Revised: The efficiency gains are particularly significant for the DAGs corresponding to real-world systems.

Original:  Introduction
Revised:  Introduction

Original: Causal models are useful for representing and reasoning about causal relationships in many areasof artificial intelligence and machine learning.
Revised: Causal models are useful for representing and reasoning about causal relationships in many areas of artificial intelligence and machine learning.

Original: One way to represent causal models is as a directedgraphical model, which consists of a causal structure in the form of directed edges between variablesand parameters in the form of a conditional probability distribution corresponding to each variable.
Revised: One way to represent causal models is as a directed graphical model, which consists of a causal structure in the form of directed edges between variables and parameters in the form of a conditional probability distribution corresponding to each variable.

Original: The edges in the directed structure convey the existence and direction of causal relationships and theparameters specify the magnitude of their causal effects.
Revised: The edges in the directed structure convey the existence and direction of causal relationships and the parameters specify the magnitude of their causal effects.

Original: To construct a causal  modelgiven a datasample, the firststep is to learn a graph structure and the second is to use that graph structure forestimating parameters.
Revised: Some approaches to constructing a causal graphical model first learn a graph structure and then use that graph structure for estimating parameters.

Original: Constraint-based algorithms are a type of structure learningalgorithm that infer a structure thatcorresponds to the conditional independence (CI) relationships within a population.
Revised: Constraint-based algorithms for structure learning infer a structure that corresponds to the conditional independence (CI) relationships within a population.

Original: These conditionalindependence relationships are inferred by performing CI tests on a data sample.
Revised: These conditional independence relationships are inferred by performing CI tests on a data sample.

Original: In this work, wefocus on constraint-based algorithms, improving their efficiency for causal structure learning.
Revised: In this work, we focus on constraint-based algorithms, improving their efficiency for causal structure learning.

Original: The phase of constraint-based algorithms that applies CI tests typically dominates the runtime of suchalgorithms, and the number of such tests in the worst case is equal to the cardinality of the powerset of the possible conditioning variables.
Revised: The phase of constraint-based algorithms that applies CI tests typically dominates the runtime of such algorithms, and the number of such tests in the worst case is equal to the cardinality of the power set of the possible conditioning variables.

Original: Specifically, constraint-based algorithms run  CI tests foreach pair of variables and possible conditioning sets, stopping only when a test infers that the pair isconditionally independent or after all feasible CI tests have been run.
Revised: Specifically, constraint-based algorithms run a CI test for each pair of variables and possible conditioning set, stopping only when a test infers that the pair is conditionally independent or after all feasible CI tests have been performed.

Original: As the number of variablesincreases, the number of CI tests  these algorithmsrun increases exponentially.
Revised: As the number of variables increases, the number of CI tests performed by these algorithms increases exponentially.

Original: Most constraint-based algorithms assume that CI tests perform more accurately on small conditioningsets.
Revised: Most constraint-based algorithms assume that CI tests perform more accurately on small conditioning sets.

Original: However, recent model-based CI tests have the potential to work effectively on very largeconditioning sets.
Revised: However, recent model-based CI tests have the potential to work effectively on very large conditioning sets.

Original: The models typically used in those tests are well-regularized, and so they canproduce accurate results even with very large numbers of conditioning variables.
Revised: The models typically used in those tests are well-regularized, and so they can produce accurate results even with very large numbers of conditioning variables.

Original: Several experimentalevaluations compare their performance with that of other CI testsand find that they perform moreaccurately thanthe older CI tests, particularly on larger conditioning sets.
Revised: Several experimental evaluations have found that model-based tests perform more accurately than older CI tests, particularly on larger conditioning sets.

Original: Substantial recent work on CI tests suggests that their accuracy and efficiency is likely to improve.
Revised: Substantial recent work on CI tests suggests that their accuracy and efficiency is likely to improve.

Original: The focus of this paper is on how such current and future tests could be used to improve the efficiencyof constraint-based algorithms for structure learning.
Revised: The focus of this paper is on how such current and future tests could be used to improve the efficiency of constraint-based algorithms for structure learning.

Original: We assume that accurate and general modelbased CI tests exist or will exist soon.
Revised: We assume that accurate and general modelbased CI tests exist or will exist soon.

Original: We explore how algorithms could make use of model-based CItests, presenting a pre-processing algorithm for the PC algorithm which relies on randomly selecting afew large conditioning sets and running model-based CI tests on those sets for every pair of variables.
Revised: We explore how algorithms could make use of model-based CI tests, presenting a pre-processing algorithm for the PC algorithm which relies on randomly selecting a few large conditioning sets and running model-based CI tests on those sets for every pair of variables.

Original: The aim of the pre-processing algorithm is to speed up the process of identifying pairs of variables thatare conditionally independent.
Revised: The aim of the pre-processing algorithm is to speed up the process of identifying pairs of variables that are conditionally independent.

Original: We provide theoretical and empirical evidence that the pre-processingalgorithm is extremely successful at this task.
Revised: We provide theoretical and empirical evidence that the pre-processing algorithm is extremely successful at this task.

Original: The core idea that makes the pre-processing algorithm useful is to perform CI tests on large conditioning sets.
Revised: The core idea that makes the pre-processing algorithm useful is to perform CI tests on large conditioning sets.

Original: Large conditioning sets are often avoided when designing algorithms for structurelearning due to a concern that such sets could easily include one or more colliders.
Revised: Large conditioning sets are often avoided when designing algorithms for structure learning due to a concern that such sets could easily include one or more colliders.

Original: Conditioning ona collider can induce dependence between a pair of variables that would otherwise be independent.
Revised: Conditioning on a collider can induce dependence between a pair of variables that would otherwise be independent.

Original: However, we show that the probability of conditioning on a collider in an otherwise unblocked path isvery small when the size of the conditioning set is large.
Revised: However, we show that the probability of conditioning on a collider in an otherwise unblocked path is very small when the size of the conditioning set is large.

Original: This enables conditioning on large variablesets to very quickly identify large numbers of conditional independence relations.
Revised: This enables conditioning on large variable sets to very quickly identify large numbers of conditional independence relations.

Original: We show that induced dependence due to conditioning on a collider is very rare when conditioningsets are large.
Revised: We show that induced dependence due to conditioning on a collider is very rare when conditioning sets are large.

Original: Further, we observe that failing to correctly infer every conditional independence inthe pre-processing step is not a major problem, because it will be caught in the next phase of thealgorithm, and that finding many CIs early greatly reduces the total number of tests.
Revised: Further, we observe that failing to correctly infer every conditional independence in the pre-processing step is not a major problem, because unrecognized CIs will be caught in the next phase of the algorithm, and that finding many CIs early greatly reduces the total number of tests.

Original: We conducta theoretical analysis and an experimental evaluation to explore the efficiency gain between the PC algorithm with pre-processing and the PC algorithm without such pre-processing.
Revised: We conduct a theoretical analysis and an experimental evaluation to explore the efficiency gain between the PC algorithm with pre-processing and the PC algorithm without such pre-processing.

Original: We conductthe experiments on both random Erd ˝ os-Renyi networks and networks that correspond to real-worldsystems.
Revised: We conduct the experiments on both random Erd ˝ os-Renyi (ER) directed acyclic graphs (DAGs) and DAGs that correspond to real-world systems.

Original: We find that the PC algorithm with pre-processing typically performs only 0.5% to 20% ofthe CI tests that the PC algorithm alone performs on networks that correspond to real-world systems.
Revised: We find that Pre-Processing Plus PC (P3PC) performs between 0.5% to 36% and often less than 10% of the CI tests that the PC algorithm alone performs on DAGs that correspond to real-world systems.

Original:  Related Work
Revised:  Related Work

Original: A wide variety of work has focused on how to improve the accuracy and efficiency of structurelearning algorithms, including work on constraint-based algorithms [1], score-based algorithms,continuous-optimization algorithms [2], and combinations of these approaches [3].
Revised: A wide variety of work has focused on how to improve the accuracy and efficiency of structure learning algorithms, including work on constraint-based algorithms [1], score-based algorithms, continuous-optimization algorithms [2], and combinations of these approaches [3].

Original: Advances inaccuracy and efficiency for continuous-optimization algorithms have been particularly prominentrecently [4], turning some structure learning into combinatorial optimization and convex optimizationproblems [5].
Revised: Advances in accuracy and efficiency for continuous-optimization algorithms have been particularly prominent recently [4], turning some structure learning into combinatorial optimization and convex optimization problems [5].

Original: Other advances in efficiency have included analyzing properties of scoring criterionssuch as BIC and AIC [6].
Revised: Other advances in efficiency have included analyzing properties of scoring criteria such as BIC and AIC [6].

Original: In contrast, in this paper we focus on improving the efficiency of constraint-based algorithms,particularly the PC algorithm
Revised: In contrast, in this paper we focus on improving the efficiency of constraint-based algorithms, particularly the PC algorithm

Original: The PC algorithm operates by performing CI tests in order ofincreasing size of conditioning sets; i.e. on empty, size 1,size 2, and larger conditioning sets,removing edges from a complete undirected graph once they discover that a conditioning set couldmake a pair of variables CI, and then orienting edge directions.
Revised: The PC algorithm performs CI tests in order of increasing size of conditioning sets (0, 1, 2, ...). The algorithm removes an edge from a complete undirected graph once it discovers that a conditioning set makes the corresponding pair of variables conditionally independent.

Original: Several avenues exist for improvingthe efficiency of the PC algorithm.
Revised: Several avenues exist for improving the efficiency of the PC algorithm.

Original: First, some work has focused on reducing the number of CItests that need to be performed [8, 9, 10, 11, 12].
Revised: First, some work has focused on reducing the number of CI tests that need to be performed [8, 9, 10, 11, 12].

Original: Second, some work has focused on improving the PC algorithm in specific applications where the general algorithm performance is weaker, such asfor partially faithful distributions in high-dimensional linear models, when the data containsheavy outliers, and when the data is ordinal [13, 14, 15, 16].
Revised: Second, some work has focused on improving the PC algorithm in specific applications where the general algorithm performance is weaker, such as for partially faithful distributions in high-dimensional linear models, when the data contains outliers, and when the data is ordinal [13, 14, 15, 16].

Original: Third, some work has focused on improvingthe hardware through GPU use  and parallelization [17, 18, 19, 9, 20, 21, 22, 23].
Revised: Third, some work has focused on improving efficiency through the use of GPUs and parallelization [17, 18, 19, 9, 20, 21, 22, 23].

Original: However, all of the above techniques for constraint-based algorithms rely on the fact that the PCalgorithm must prioritize performing CI tests with small conditioning sets.
Revised: However, all of the above techniques for constraint-based algorithms rely on the fact that the PC algorithm must prioritize performing CI tests with small conditioning sets.

Original: The literature suggeststhat the older CI tests do not perform well on larger conditioning sets and that mistakes early on in the
Revised: The literature suggests that the older CI tests do not perform well on larger conditioning sets and that mistakes early on in the

Original: PC algorithm severely affect the structure learned by the PC algorithm since early errors propagateforward [7].
Revised: PC algorithm severely affect the structure learned by the PC algorithm since early errors propagate forward [7].

Original: Although older CI tests may not be very accurate specifically on larger conditioning sets, newconditional independence tests continue to be developed.
Revised: Although older CI tests are typically inaccurate given large conditioning sets, new conditional independence tests continue to be developed.

Original: Some new tests are model-based [24],which do not suffer from the issue of only performing well on smaller conditioning set sizes.
Revised: Some new tests are model-based [24], and perform far better on large conditioning sets.

Original: Oneexample is the Classifier Conditional Independence Test (CCIT) which uses the conditioning set suchthat its variables are predictor variables for a model [25].
Revised: One example is the Classifier Conditional Independence Test (CCIT), which constructs classification models using the conditioning set and other variables to test conditional independence [25].

Original: In such a case, the CI test using a largeconditioning setmay not necessarily produce less accurate results than that using a small conditioningset.
Revised: Modern CI tests such as CCIT and related methods are far less sensitive to the size of the conditioning set than more conventional tests.

Original: Further, many CI tests are fast and are continuously improved to be faster.
Revised: Further, many CI tests are fast and are becoming faster.

Original: Some very quick CItests such as the Randomized Conditional Independence Test (RCIT) [26] are improvements to CItests that were too slow to be applied such as the Kernel-Based Conditional Independence Test (KCIT)[27].
Revised: Some very fast CI tests such as the Randomized Conditional Independence Test (RCIT) [26] are improvements to CI tests that were too slow to be applied such as the Kernel-Based Conditional Independence Test (KCIT) [27].

Original: Other CI tests such as CCIT are not almost instantaneous, but also not intractable.
Revised: Other CI tests such as CCIT are not almost instantaneous, but also not intractable.

Original: Further,interest has even extended to conditional dependence measures, which provide a more accurateestimate of the extent of conditional dependence.

Original: Given these new improvements and work, we couldassume that even if a CI test does not meet a standard of quickness and accuracy, such a test will bedeveloped in the near future.
Revised: Given these new improvements and work, we could assume that even if a CI test does not meet a standard of quickness and accuracy, such a test will be developed in the near future.

Original:  Pre-processing Algorithm
Revised:  Pre-processing Algorithm

Original: We propose a pre-processing algorithm for the PC algorithm that leverages the use of CI tests thatperform accurately and efficiently on large conditioning sets:
Revised: We propose a pre-processing algorithm for the PC algorithm that leverages the use of CI tests which perform accurately and efficiently on large conditioning sets:

Original: Input : X = [ x 1 , x 2 , , x n ] : a dataset where x i is a vector containing data samples for the i th variable and n is the number of variables in the dataset; V : the set of variables 
Revised: Input : X = [ x 1 , x 2 , , x n ] : a dataset where x i is a vector containing data samples for the i th variable and n is the number of variables in the dataset; V : the set of variables Output : p

Original: Output : p = [ v 1 , v 2 , . .
Revised: = [ v 1 , v 2 , . .

Original: . , v n ] : an n × n matrix where index [ i, j ] is 0 if the pair of variables could be made conditionally independent and 1 otherwise a Set p [ a b p [ b a ]
Revised: . , v n ] : an n × n matrix where index [ i, j ] is 0 if the pair of variables could be made conditionally independent and 1 otherwise 1 Initialize p as an n × n matrix

Original: = go to the next iteration the for loop for i ∈ { 1 , . .

Original: , c 1 } do

Original: Append a random set of n − c 2 variables not including a and b to list L for S ∈ L do if a ⊥ b | S then

Original: [-Set p [ a, b ] = 
-]
Revised: [-Set p [ a, b ] = 
-]

Original: and p [ b, a ] = 0 ; go to the next iteration of the for loop 11 Return p

Original: The pre-processing algorithm aims to inform the PC algorithm aboutpairs which could possibly bemade conditionally independent, before it runs.
Revised: The pre-processing algorithm can be used to inform the PC algorithm about conditionally independent variable pairs before PC runs.

Original: Such knowledge decreases the  number of CI teststhat the PCalgorithm performs as indicated by Section 4.
Revised: Such knowledge decreases the total number of CI tests that PC performs (see details in Sections 4 and 5).

Original: For each pair of variables, the algorithmrandomly selects c 1 conditioning sets of size n − c 2 where c 1 and c 2 are both near-zero integers.
Revised: For each pair of variables, the algorithm randomly selects c 1 conditioning sets of size n − c 2 where c 1 and c 2 are both small positive integers.

Original: Forthe derivations in Section 4, we set c 1 = 3 and c 2 = 4 .
Revised: For the derivations in Section 4, we set c 1 = 3 and c 2 = 4 .

Original: Next, the algorithm aims to perform CI testson the empty set and on each of the c 1 conditioning sets to determineif we could tell whether a paircould be made conditionally independent.
Revised: Next, the algorithm performs a CI test using the empty set and on each of the c 1 conditioning sets to determine whether the variable pair is conditionally independent.

Original: The pre-processing algorithm we propose relies on tests that are accurate on large conditioning sets(e.g., a model-based CI test).
Revised: The pre-processing algorithm we propose relies on tests that are accurate on large conditioning sets (e.g., a model-based CI test).

Original: However, after pre-processing, the subsequent application of the PCalgorithm still uses traditional CI tests.
Revised: However, after pre-processing, the subsequent application of the PC algorithm still uses traditional CI tests.

Original: Although fast model-based CI tests exist, given the currentuncertain speed of model-based CI tests in comparison to traditional CI tests, the most efficientalgorithm is likely to use model-based CI tests only where necessary, which is when the conditioningset size is large (in the pre-processing algorithm)
Revised: Although fast model-based CI tests exist, given the current relative speed of model-based and traditional CI tests, the most efficient algorithm is likely to use model-based CI tests only for pre-processing.

Original:  Theoretical Analysis for Erd˝os-Renyi Networks
Revised:  Theoretical Analysis for Erd˝os-Renyi DAGs

Original: Definitions.
Revised: Definitions.

Original: We define a trail of length ℓ between nodes u 0 and u ℓ as a sequence of ℓ distinct edgese 1 , . . .
Revised: We define a trail of length ℓ between nodes u 0 and u ℓ as a sequence of ℓ distinct edges e 1 , . . .

Original: , e ℓ where u 0 is an endpoint of e 1 , u ℓ is an endpoint of e ℓ , and each e i has one endpoint incommon with e i − 1 and the other endpoint in common with e i +1 .
Revised: , e ℓ where u 0 is an endpoint of e 1 , u ℓ is an endpoint of e ℓ , and each e i has one endpoint in common with e i − 1 and the other endpoint in common with e i +1 .

Original: Note that the head (or “child” node)of e i − 1 does not need to equal the tail (or “parent” node) of e i .
Revised: Note that the head (or “child” node) of e i − 1 does not need to equal the tail (or “parent” node) of e i .

Original: The node in common with e i ande i +1 is a collider in the trail if it is the head of both the edges e i and e i +1 .
Revised: The node in common with e i and e i +1 is a collider in the trail if it is the head of both the edges e i and e i +1 .

Original: A trail is blocked by aconditioning set if at least one intermediate node on the trail that is not a collider is in the conditioningset or if at least one intermediate node on the trail that is a collider is not in the conditioning set.
Revised: A trail is blocked by a conditioning set if at least one intermediate node on the trail that is not a collider is in the conditioning set or if at least one intermediate node on the trail that is a collider is not in the conditioning set.

Original: Apair of variables is made CI if all trails between the pair of variables are blocked by a conditioningset.
Revised: A pair of variables is inferred to be conditionally independent if all trails between the pair of variables are blocked.

Original: The goal of the pre-processing algorithm is to detect if a pair of variables could be made CI ornot.
Revised: The goal of the pre-processing algorithm is to detect whether a pair of variables can be inferred to be conditionally independent.

Original: Erd˝os-Renyi networks.
Revised: Erd˝os-Renyi DAGs.

Original: For Erd˝os-Renyi networks where n nodes are labeled in increasing orderand an edge exists with probability p  only  from nodes with a smaller label to nodes with a larger label,we provide indications that one randomly selected conditioning set of size n − 4 would be able todetermine whether a pair of variables could possibly be made CI.
Revised: An Erd˝os-Renyi (ER) DAG on n nodes is randomly generated as follows: the n nodes are labeled in increasing order and an edge exists with probability p and can only have a direction from nodes with a smaller label to nodes with a larger label. For ER DAGS, we provide indications that one randomly selected conditioning set of size n − 4 would be able to determine whether a pair of variables is CI. We provide the theoretical analysis with respect to conditioning sets specifically of size n − 4 despite being able to generalize the result to conditioning sets of size n − c 2 for 2 ≤ c 2 ≤ n because n − 4 is guaranteed to be a reasonably large conditioning set size for most DAGs. Thus, we could more easily examine the effect of a large conditioning set size on the process of determining whether a pair of variables could possibly be made CI.

Original: We index the nodes v 1 , v 2 , . .
Revised: We index the nodes v 1 , v 2 , . .

Original: . , v nwhere v i is the node with the i th smallest label.
Revised: . , v n where v i is the node with the i th smallest label.

Original: If a pair of variables could be detected as CI with respect to a conditioning set of size n − 4 , thenall possible trails between the pair of variables are blocked with respect to the conditioning set.
Revised: If a pair of variables could be detected as CI with respect to a conditioning set of size n − 4 , then all possible trails between the pair of variables are blocked with respect to the conditioning set.

Original: Weindicate that the probability that all possible trails between a pair of variables are blocked with respectto the conditioning set is low by considering trails of length 7 and more and trails of length 6 andfewer separately.
Revised: We indicate that the probability that all possible trails between a pair of variables are blocked with respect to the conditioning set is low by considering trails of length 7 and more and trails of length 6 and fewer separately.

Revised: The specific raw values of 7 and 6 rely on the fixed large conditioning set size n − 4 .

Original: Statement 1 (Trails of length at least 7 ).
Revised: Statement 1 (Trails of length at least 7 ).

Original: A conditioning set of size n − 4 will block every trail oflength 7 or greater.
Revised: A conditioning set of size n − 4 will block every trail of length 7 or greater.

Original: Note that this will be true for any network, not just Erd ˝ os-Renyi networks.
Revised: Note that this will be true for any DAG, not just Erd˝os-Renyi DAGs.

Original: The reason for this is as follows.
Revised: The reason for this is as follows.

Original: First note that for any trail of length at least 7 , there are at least 3intermediate nodes on the trail that are not colliders; this follows because  colliders cannot be adjacentto each other and therefore if there are k intermediate nodes, at least ⌊ k/ 2 ⌋ of them are non-colliders.
Revised: First note that for any trail of length at least 7 , there are at leastintermediate nodes on the trail that are not colliders; this follows because assuming that all edges are unidirectional, colliders cannot be adjacent to each other and therefore if there are k intermediate nodes, at least ⌊ k/ 2 ⌋ of them are non-colliders.

Original: Next consider a trail of length at least 7 between a pair of nodes a and b and a conditioning set S ∈ { v 1 , . .
Revised: Next consider a trail of length at least 7 between a pair of nodes a and b and a conditioning set S ∈ { v 1 , . .

Original: . , v n } \ { a, b } of size n − 4 .
Revised: . , v n } \ { a, b } of size n − 4 .

Original: Since { v 1 , . . .
Revised: Since { v 1 , . . .

Original: , v n } \ ( S ∪ { a, b } ) only has size 2 , it must bethat S contains at least one of the three or more non-colliders on the trail and hence blocks the trail.
Revised: , v n } \ ( S ∪ { a, b } ) only has size 2 , it must be that S contains at least one of the three or more non-colliders on the trail and hence blocks the trail.

Original: Statement 2 (Trails of length at most 6 ).
Revised: Statement 2 (Trails of length at most 6 ).

Original: In an Erd˝os-Renyi network, the expected number of trailsof length ≤ 6 between a specific pair of nodes is at most p (1 + ( pn ) + ( pn ) 2 + . . .
Revised: In an Erd ˝ os-Renyi DAG, the expected number of trails of length ≤ 6 between a specific pair of nodes is at most p (1+( pn )+( pn ) 2 + . . .

Original: ( pn ) 6 ) ≤ 7 n 6 p 7where the last inequality assumes p ≥ 1 /n .
Revised: ( pn ) 6 ) ≤ 7 n 6 p 7 where the last inequality assumes p ≥ 1 /n .

Original: If p = Θ(1) /n , corresponding to graphs with constantexpected degree, this expectation tends to 0 as n → ∞ and hence, by the Markov bound, theprobability there exists such a trail tends to 0 as n → ∞ .
Revised: If p = Θ(1) /n , corresponding to graphs with constant expected degree, this expectation tends to 0 as n → ∞ and hence, by the Markov bound, the probability there exists such a trail tends to 0 as n → ∞ .

Original: This statement can be proved as follows.
Revised: This statement can be proved as follows.

Original: There are (cid:0) n − 2 k (cid:1) choices for the set of k intermediate nodeson a trail of length k + 1 , there are k ! orderings of these nodes, and the probability there existsedges between each successive pair of nodes is p k +1 .
Revised: There are (cid:0) n − 2 k (cid:1) choices for the set of k intermediate nodes on a trail of length k + 1 , there are k ! orderings of these nodes, and the probability there exists edges between each successive pair of nodes is p k +1 .

Original: Hence, the expected number of trails with kintermediate nodes is k !
Revised: Hence, the expected number of trails with k intermediate nodes is k !

Original: (cid:0) n − 2 k (cid:1)
Revised: (cid:0) n − 2 k (cid:1)

Original: p k +1 < n k p k +1 .
Revised: p k +1 < n k p k +1 .

Original: Summing from k = 0 to 6 gives the claimed result.
Revised: Summing from k = 0 to 6 gives the claimed result.

Original: Finally we show that we expect most nodes to not be colliders for any trail.
Revised: Finally we show that we expect most nodes to not be colliders for any trail.

Original: This implies that evenwhen there are short trails, they will likely contain more non-colliders than colliders.
Revised: This implies that even when there are short trails, they will likely contain more non-colliders than colliders.

Original: And if theconditioning set includes at least one non-collider in a trail, then that trail is blocked.
Revised: And if the conditioning set includes at least one non-collider in a trail, then that trail is blocked.

Original: Statement 3 (Expected number of colliders).
Revised: Statement 3 (Expected number of colliders).

Original: The expected number of nodes that are colliders forsome trail is
Revised: The expected number of nodes that are colliders for some trail is

Original: [Equation]
Revised: [Equation]

Original: For example, when p = 1 /n , the expression corresponds to 0 .
Revised: For example, when p = 1 /n , the expression corresponds to 0 .

Original: 104 n .
Revised: 104 n .

Original: This is because the probability that v i , the node with the i th largest label, is a collider is
Revised: This is because the probability that v i , the node with the i th largest label, is a collider is

Original: [Equation]
Revised: [Equation]

Original: since the only way for a node v i to be a collider on any trail is if are edges between it and two nodesof more nodes amongst { v 1 , . . .
Revised: since the only way for a node v i to be a collider on any trail is if are edges between it and two nodes of more nodes amongst { v 1 , . . .

Original: v i − 1 } .
Revised: v i − 1 } .

Original: Using linearity of expectation, the expected number of nodesthat are colliders for some trail is (
Revised: Using linearity of expectation, the expected number of nodes that are colliders for some trail is (cid:80) ni =3 P

Original: cid:80) ni =3 P ( Bin ( i − 1 , p ) ≥ 2) and this simplifies to Eq.
Revised: ( Bin ( i − 1 , p ) ≥ 2) and this simplifies to Eq.

Original: Takeaway.
Revised: Takeaway.

Original: For a single pair of variables, one unblocked trail will prevent the pre-processing algorithmfrom correctly determining that the pair could be made CI.
Revised: For a single pair of variables, one unblocked trail will prevent the pre-processing algorithm from correctly determining that the pair could be made CI.

Original: Yet, we show that one unblocked trail of length 7 or more is impossible, that the expected number of trails of length 2 , 3 , 4 , 5 , or 6 is verysmall, and that the expected number of nodes that are colliders for a trail is very small.
Revised: Yet, we show that one unblocked trail of length 7 or more is impossible, that the expected number of trails of length 2 , 3 , 4 , 5 , or 6 is very small, and that the expected number of nodes that are colliders for a trail is very small.

Original: Since theexpected number of short trails and of colliders is small, the large conditioning sets will very likelycatch some pairs that may take manyCI tests to be concluded CI using the original PC algorithm.
Revised: Since the expected number of short trails and of colliders is small, the large conditioning sets will very likely catch some pairs that would otherwise require many tests before the original PC algorithm would declare them to be conditionally independent.

Original:  Experiments
Revised:  Experiments

Original: We performed experiments to estimate the raw number of CI tests performed by  the  PC algorithm withour pre-processing algorithm and without it.
Revised: We performed experiments to estimate the raw number of CI tests performed by P3PC and by the original PC algorithm.

Original: Our goal was to evaluate the effects of pre-processingon the total number of tests run by the algorithms.
Revised: Our goal was to evaluate the effects of pre-processing on the total number of tests run by the algorithms.

Original: Thus, we conducted experiments using known DAG structure and standard rules of d-separation to determine whether variables could be madeconditionally independent by a given conditioning set, rather than running conditional independencetests on data generated by that DAG.
Revised: Thus, we conducted experiments using known DAG structure and standard rules of d-separation to determine whether variables could be made conditionally independent by a given conditioning set, rather than running conditional independence tests on data generated by that DAG.

Revised: lllllllllllllllllllllllllllllllllllllllllllllll l llllllllllll l lllllllllllllllllllll l llllllllllllllllllllllll l lllllll l llllllllllllllllllllllllll l l l l llll l l lll l lllllllllllllllllll l llllll l l l l l l l l llllllllllllllllllllllllllllll l l lll l l l l l l l l llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l l lll l l ll l ll l l lll l l llllllllllllllllllllllllll l lllllllllllllllllllllllllllll l l l l l l l l ll l lll l l l l lll l l l l l l l ll ll l l l l l l l l l l l l l ll l l l l l l l l l l l l ll ll l l lll l lll l ll ll l lllll l llll ll ll l l ll ll l l l l lll l lll l l ll l l lll l lllllllllllllllll l lllllllll l llllllllllllllll l l l l l l l l l l l l l ll llllllllllllllllllllllllllllllllllllllllllll

Revised: l l l l l l ll l l l l l l l l l l l l l l l lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l ll l l l l lllll l l lllll ll ll ll l l lll

Original:  The first set of experiments compared the raw number of tests performed by the PC algorithm withand without pre-processing on over 1,000 Erd˝os-Renyi networks randomly generated in the mannerdescribed in Section 4.
Revised: l l l l l l l l [Figure] The first set of experiments compared the raw number of tests performed by the PC algorithm with and without pre-processing on over 1,000 Erd˝os-Renyi DAGs randomly generated in the manner described in Section 4.

Original: The second set of experiments compared the same two algorithms on networksin the bnlearn repository, which are designed to correspond to real-world systems.
Revised: The second set of experiments compared the same two algorithms on DAGs in the bnlearn repository, which are designed to correspond to real-world systems.

Original: For example, the Water network is from a technical report on an expert system for controlling waste water treatment[28].
Revised: For example, the Water DAG is from a technical report on an expert system for controlling waste water treatment [28].

Original: The number of nodes in the Erd ˝ os-Renyi networks is smaller than that of the realistic networksin the bnlearn repository because we aimed to perform experiments on a large number of ERnetworks and inferring d-separation is time-consuming on larger, dense 
Revised: The number of nodes in the Erd˝os-Renyi DAGs is smaller than that of the realistic DAGs in the bnlearn repository because we aimed to perform experiments on a large number of ER DAGs and inferring d-separation is time-consuming on larger, dense DAGs.

Original: networks.lllllllllllllllllllllllllllllllllllllllllllllll l llllllllllll l lllllllllllllllllllll l llllllllllllllllllllllll l lllllll l llllllllllllllllllllllllll l l l l llll l l lll l lllllllllllllllllll l llllll l l l l l l l l llllllllllllllllllllllllllllll l l lll l l l l l l l l llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l l l ll l l ll l ll l l lll l l llllllllllllllllllllllllll l lllllllllllllllllllllllllllll l l l l l l l l ll l lll l l l l lll l l l l l l l ll ll l l l l l l l l l l l l l l l l l l l l l l l l l l l ll ll l l lll l lll l ll ll l l llll l llll ll ll l l ll ll l l l l lll l lll l l ll l l lll l lllllllllllllllll l lllllllll l llllllllllllllll l l l l l l l l l l l l l ll llllllllllllllllllllllllllllllllllllllllllll l l l l l l ll l l l l l l l l l l l l l l l lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll l ll l l l l lllll l l lllll ll ll ll l l lll
Revised: In both the DAGs that correspond to real-world systems and the Erd˝os-Renyi DAGs, the raw number of tests performed by P3PC is much smaller than those performed by the PC algorithm alone as shown in Figure 1. The efficiency gain is greater as the number of edges increases to be twice the amount of the number of nodes.

Original: l l l l l l l l l l l [Figure] Preprocessing
Revised: The difference in performance of P3PC and the PC algorithm alone is more noticeable for the DAGs that correspond to real-world systems, where the number of tests performed by P3PC is so much smaller than that performed by the PC algorithm alone that the bar representing that number often is barely visible in Figure 2.

Original: [Figure]
Revised: Specifically, for the DAGs that correspond to real-world systems, the number of tests performed by P3PC is between 0.5% to 36% and often less than 10% of the number of tests performed by the PC algorithm alone as displayed in

Original: [Table] Table 
Revised: Table 1. [Table]

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a pre-processing algorithm for the PC algorithm which can dramatically reduce the totalnumber of CI tests that the PC algorithm performs.
Revised: We proposed a pre-processing algorithm for the PC algorithm which can dramatically reduce the total number of CI tests that the PC algorithm performs.

Original: This implies a new reason to prioritize researchon accurate model-based tests of conditional independence, particularly those that perform wellwith large conditioning sets.
Revised: This implies a new reason to prioritize research on accurate model-based tests of conditional independence, particularly those that perform well with large conditioning sets.

Original: Furthermore, we show theoretically why the pre-processing algorithmreduces this number in expectation for Erd˝os-Renyi graphs.
Revised: Furthermore, we show theoretically why the pre-processing algorithm reduces this number in expectation for Erd ˝ os-Renyi graphs.

Original: Finally, we perform experiments onnetworks that correspond to real-world systems, finding that the number of tests performed by the PCalgorithm with pre-processing is typically 0.5% to 20% of the number of tests performed by the PCalgorithm alone.
Revised: Finally, we perform experiments on DAGs that correspond to real-world systems, finding that the number of tests performed by P3PC is between 0.5% to 36% and often less than 10% of the number of tests performed by the PC algorithm alone.

Original: From the experiments on Erd ˝ os-Renyi graphs, we observe that the efficiency gain isalso large and increases as the number of edges increases to be twice the number of nodes.
Revised: From the experiments on Erd˝os-Renyi graphs, we observe that the efficiency gain is also large and increases as the number of edges increases to be twice the number of nodes.

Original: This work addresses one of the primary challenges for routine use of the PC algorithm by greatlyreducing runtime.
Revised: This work addresses one of the primary challenges for routine use of the PC algorithm by greatly reducing runtime.

Original: This makes possible improved algorithms such as PC-stable, which repeats thesteps in the PC algorithm many times, more tractable [29].
Revised: This makes possible improved algorithms such as PC-stable, which repeats the steps in the PC algorithm many times, more tractable [29].

Original: To further increase the efficiency of thepre-processing step, a system could parallelize the tests within the pre-processing algorithm for eachpair since the tests for different pairs do not depend on each other. 
Revised: To further increase the efficiency of the pre-processing step, a system could parallelize the tests within the pre-processing algorithm for each pair since the tests for different pairs do not depend on each other. Acknowledgments

Revised: This research was sponsored by the National Science Foundation, the Defense Advanced Research

Revised: Projects Agency (DARPA), the United States Air Force, and the Army Research Office (ARO) under grants NSF CCF-1934846 and NSF CCF-1908849, Contract Number FA8750-17-C-0120, and

Revised: Cooperative Agreements Number W911NF-20-2-0005 and W911NF-20-2-0005.

Revised: The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of NSF, DARPA, the USAF, ARO, or the U.S. Government.

Revised: The U

Revised: S. Government is authorized to reproduce and distribute reprints for

Revised: Government purposes not withstanding any copyright notation herein.