Original: O RDERING -B ASED C AUSAL D ISCOVERY WITH R EINFORCEMENT L EARNING
Revised: O RDERING -B ASED C AUSAL D ISCOVERY WITH R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: It is a long-standing question to discover causal relations among a set of variables in many empirical sciences.
Revised: It is a long-standing question to discover causal relations among a set of variables in many empirical sciences.

Original: Recently, Reinforcement Learning (RL) has achieved promising results in causal discovery.
Revised: Recently, Reinforcement Learning (RL) has achieved promising results in causal discovery.

Original: However, searching the space of directed graphs directly and enforcing acyclicity by implicit penalties tend to be inefﬁcient and restrict the method tothe small problems.
Revised: However, searching the space of directed graphs directly and enforcing acyclicity by implicit penalties tend to be inefﬁcient and restrict the method to small problems.

Original: In this work, we alternatively consider searching an ordering by RL from the variable ordering space that is much smaller than that of directed graphs, which also helps avoid dealing with acyclicity.
Revised: In this work, we alternatively consider searching an ordering by RL from the variable ordering space that is much smaller than that of directed graphs, which also helps avoid dealing with acyclicity.

Original: Speciﬁcally, we formulate the ordering search problem as a Markov decision process, and then use different reward designs to optimize the ordering generating model.
Revised: Specifically, we formulate the ordering search problem as a Markov decision process, and then use different reward designs to optimize the ordering generating model.

Original: A generated ordering is then processed using variable selection methods to obtain the ﬁnal directed acyclic graph.
Revised: A generated ordering is then processed using variable selection methods to obtain the ﬁnal directed acyclic graph.

Original: In contrast to other causal discovery methods, our method can also utilize a pretrained model to accelerate training.
Revised: In contrast to other causal discovery methods, our method can also utilize a pretrained model to accelerate training.

Original: We conduct experiments on both synthetic and real-world datasets, and show that the proposed method outperforms other baselines on important metrics even on large graph tasks.
Revised: We conduct experiments on both synthetic and real-world datasets, and show that the proposed method outperforms other baselines on important metrics even on large graph tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Identifying causal structure from observational data is an important but also challenging task in many applications.
Revised: Identifying causal structure from observational data is an important but also challenging task in many applications.

Original: This problem can be formulated as that of ﬁnding a Directed Acyclic Graph (DAG) that minimizes some score function deﬁned w.r.t.
Revised: This problem can be formulated as that of ﬁnding a Directed Acyclic Graph (DAG) that minimizes some score function deﬁned w.r.t.

Original: observed data.
Revised: observed data.

Original: Though there exist well-studied score functions like Bayesian Information Criterion (BIC) or Minimum Description Length (MDL) (Schwarz et al., 1978; Chickering, 2002), searching over the space of DAGs is known to be NP-hard, even if each node has at most two parents (Chickering, 1996).
Revised: Though there exist well-studied score functions like Bayesian Information Criterion (BIC) or Minimum Description Length (MDL) (Schwarz et al., 1978; Chickering, 2002), searching over the space of DAGs is known to be NP-hard, even if each node has at most two parents (Chickering, 1996).

Original: Consequently, traditional methods mostly rely on local heuristics to perform the search, including greedy hill climbing and Greedy
Revised: Consequently, traditional methods mostly rely on local heuristics to perform the search, including greedy hill climbing and Greedy

Original: Equivalence Search (GES) that explore the Markov equivalence classes (Chickering, 1996).
Revised: Equivalence Search (GES) that explore the Markov equivalence classes (Chickering, 1996).

Original: Along with various search strategies, existing methods have also considered to reduce the search space while meeting the DAG constraint.
Revised: Along with various search strategies, existing methods have also considered to reduce the search space while meeting the DAG constraint.

Original: A useful practice is to cast the causal structure learning problem as that of learning an optimal ordering of variables (Koller & Friedman, 2009).
Revised: A useful practice is to cast the causal structure learning problem as that of learning an optimal ordering of variables (Koller & Friedman, 2009).

Original: Because the ordering space is signiﬁcantly smaller than the space of directed graphs and searching over ordering space can avoid the problem of dealing with acyclic constraints (Teyssier & Koller, 2005).
Revised: Because the ordering space is signiﬁcantly smaller than the space of directed graphs and searching over ordering space can avoid the problem of dealing with acyclic constraints (Teyssier & Koller, 2005).

Original: Many algorithms are used to search for ordering such as genetic algorithm (Larranaga et al., 1996), Markov chain Monte Carlo (Friedman & Koller, 2003) and greedy local hill-climbing (Teyssier & Koller, 2005).
Revised: Many algorithms are used to search for ordering such as genetic algorithm (Larranaga et al., 1996), Markov chain Monte Carlo (Friedman & Koller, 2003) and greedy local hill-climbing (Teyssier & Koller, 2005).

Original: However, these algorithms often cannot ﬁnd the best ordering effectively.
Revised: However, these algorithms often cannot ﬁnd the best ordering effectively.

Original: Recently, with differentiable score functions, several gradient-based methods have been proposed based on a smooth characterization of acyclicity, including NOTEARS (Zheng et al., 2018) for linear causal models and several subsequent works, e.g., Yu et al.
Revised: Recently, with differentiable score functions, several gradient-based methods have been proposed based on a smooth characterization of acyclicity, including NOTEARS (Zheng et al., 2018) for linear causal models and several subsequent works, e.g., Yu et al.

Original: ; Lachapelle et al.
Revised: ; Lachapelle et al.

Original: (2020); Ng et al.
Revised: (2020); Ng et al.

Original: (2019b;a), which use neural networks to model nonlinear causal relationships.
Revised: (2019b;a), which use neural networks to model nonlinear causal relationships.

Original: As another attempt,
Revised: As another attempt,

Original: Zhu et al.
Revised: Zhu et al.

Original: (2020) utilize Reinforcement Learning (RL) as a search strategy to ﬁnd the best DAG from the graph space and it can be incorporated with a wide range of score functions.
Revised: (2020) utilize Reinforcement Learning (RL) as a search strategy to ﬁnd the best DAG from the graph space and it can be incorporated with a wide range of score functions.

Original: Unfortunately, its good performance is achieved only with a moderate number of variables ( 30 variables or so), for at least two reasons: 1) the action space, consisting of directed graphs, is tremendous for large scale problems and hard to be explored efﬁciently; and 2) it has to compute scores for many non-DAGs generated during training but computing scores w.r.t.
Revised: Unfortunately, its good performance is achieved only with around 30 variables, for at least two reasons: 1) the action space, consisting of directed graphs, is tremendous for large scale problems and hard to be explored efﬁciently; and 2) it has to compute scores for many non-DAGs generated during training but computing scores w.r.t.

Original: data is generally be time-consuming.
Revised: data is generally be time-consuming.

Original: It appears that the RL-based approach may not be able to achieve a close performance to other gradient-based methods that directly optimize the same (differentiable) score function for large causal discovery problems, due to its search nature.
Revised: It appears that the RL-based approach may not be able to achieve a close performance to other gradient-based methods that directly optimize the same (differentiable) score function for large causal discovery problems, due to its search nature.

Original: In this work, we propose a RL-based approach for causal discovery, named Causal discovery with
Revised: In this work, we propose a RL-based approach for causal discovery, named Causal discovery with

Original: Ordering-based Reinforcement Learning (CORL), which combines RL with the ordering based paradigm so that we can exploit the powerful search ability of RL to search the best ordering efﬁciently.
Revised: Ordering-based Reinforcement Learning (CORL), which combines RL with the ordering based paradigm so that we can exploit the powerful search ability of RL to search the best ordering efﬁciently.

Original: To achieve this, we formulate the ordering search problem as a Markov Decision Process (MDP), and then use different reward designs for RL to optimize the ordering generating model.
Revised: To achieve this, we formulate the ordering search problem as a Markov Decision Process (MDP), and then use different reward designs for RL to optimize the ordering generating model.

Original: In addition, we notice that pretrained model can be incorporated into our model to accelerate training.
Revised: In addition, we notice that pretrained model can be incorporated into our model to accelerate training.

Original: For a generated ordering, we prune it to the ﬁnal DAG by variable selection.
Revised: For a generated ordering, we prune it to the ﬁnal DAG by variable selection.

Original: The proposed approach is evaluated on both synthetic and real datasets to validate its effectiveness.
Revised: The proposed approach is evaluated on both synthetic and real datasets to validate its effectiveness.

Original: In particular, the proposed method can achieve a much improved performance than the previous RL-based method on both linear and non-linear data models, even outperforms NOTEARS, a gradient-based method, on 150 -node linear data models, and is competitive with Causal Additive Model (CAM) on non-linear data models.
Revised: In particular, the proposed method can achieve a much improved performance than the previous RL-based method on both linear and non-linear data models, even outperforms NOTEARS, a gradient-based method, on 150 -node linear data models, and is competitive with Causal Additive Model (CAM) on non-linear data models.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Existing causal discovery methods roughly fall into three classes.
Revised: Existing causal discovery methods roughly fall into three classes.

Original: The ﬁrst class, as described in the introduction, are the so-called score-based methods.
Revised: The ﬁrst class, as described in the introduction, are the so-called score-based methods.

Original: Besides the mentioned BIC/MDL scores, other score functions include the Bayesian Gaussian equivalent score (Geiger & Heckerman, 1994), the generalized score based on (conditional) independence relationship (Huang et al., 2018), and a recently proposed meta-transfer score (Bengio et al., 2020).
Revised: Besides the mentioned BIC/MDL scores, other score functions include the Bayesian Gaussian equivalent score (Geiger & Heckerman, 1994), the generalized score based on (conditional) independence relationship (Huang et al., 2018), and a recently proposed meta-transfer score (Bengio et al., 2020).

Original: Another class of methods such as fast causal inference and PC (Spirtes et al., 2000; Zhang, 2008) which ﬁrst ﬁnd causal skeleton and then decide the orientations of the edges up to the Markov equivalence class, are viewed as constraint-based.
Revised: Another class of methods such as fast causal inference and PC (Spirtes et al., 2000; Zhang, 2008) which ﬁrst ﬁnd causal skeleton and then decide the orientations of the edges up to the Markov equivalence class, are viewed as constraint-based.

Original: Such methods usually involve multiple independent testing problems; the testing results may have conﬂicts and handling them is not easy.
Revised: Such methods usually involve multiple independent testing problems; the testing results may have conﬂicts and handling them is not easy.

Original: The last class  methods based on properly deﬁned functional causal models, include Linear Non-Gaussian Acyclic Model (LiNGAM), nonlinear Additive Noise
Revised: The last class of methods relies on properly deﬁned functional causal models, includeing Linear Non-Gaussian Acyclic Model (LiNGAM), nonlinear Additive Noise

Original: Model (ANM) (Hoyer et al., 2009; Peters et al., 2014), and the post-nonlinear causal model (Zhang & Hyvärinen, 2009).
Revised: Model (ANM) (Hoyer et al., 2009; Peters et al., 2014), and the post-nonlinear causal model (Zhang & Hyvärinen, 2009).

Original: By placing certain assumptions on the class of causal functions and/or noise distributions, these methods can distinguish different DAGs in the same Markov equivalence class.
Revised: By placing certain assumptions on the class of causal functions and/or noise distributions, these methods can distinguish different DAGs in the same Markov equivalence class.

Original: Since we particularly consider ordering-based approaches, here we present a more detailed review of such methods.
Revised: Since we particularly consider ordering-based approaches, here we present a more detailed review of such methods.

Original: Most of the ordering-based methods belong to the class of score-based methods.
Revised: Most of the ordering-based methods belong to the class of score-based methods.

Original: The score of an ordering is usually deﬁned as the score of the best DAG that is consistent with the given ordering.

Original: Besides the mentioned heuristic search algorithms, Schmidt et al.
Revised: Besides the mentioned heuristic search algorithms, Schmidt et al.

Original: (2007) proposed L1OBS to conduct variable selection using (cid:96) 1 -regularization paths based on Teyssier & Koller (2005).
Revised: (2007) proposed L1OBS to conduct variable selection using (cid:96) 1 -regularization paths based on Teyssier & Koller (2005).

Original: Scanagatta et al.
Revised: Scanagatta et al.

Original: (2015) further proposed an ordering exploration method on the basis of an approximated score function so as to scale to thousands of variables.
Revised: (2015) further proposed an ordering exploration method on the basis of an approximated score function so as to scale to thousands of variables.

Original: For nonlinear data models, the Causal Additive Model (CAM)based ordering search method was developed by Bühlmann et al. 
Revised: The Causal Additive Model (CAM) was developed by Bühlmann et al. (2014) for nonlinear data models.

Revised: Some recent ordering-based methods such as sparsest permutation (Raskutti & Uhler, 2018) and greedy sparest permutation (Solus et al., 2017) can guarantee consistency of Markov equivalence class, based on some conditional independence relations and certain assumptions like faithfulness.

Revised: A variant of greedy sparest permutation was also proposed in Bernstein et al.

Original:  In addition, several exact algorithms such as Dynamic Programming (DP) (Silander & Myllymäki, 2006;Perrier et al., 2008) and integer or linear programming (Jaakkola et al., 2010;
Revised: (2020) for the setting with latent variables. In the present work, we mainly work on identiﬁable cases which may have different assumptions from theirs. In addition, several exact algorithms such as dynamic programming (Silander & Myllymäki, 2006;

Original:  Cussens, 2011; Bartlett & Cussens, 2017) are used to solvethe causal discovery problem.
Revised: Perrier et al., 2008) and integer or linear programming (Jaakkola et al., 2010; Cussens, 2011; Bartlett & Cussens, 2017) are used to solve causal discovery problem.

Original: However, these exact algorithms usually work on small graphs with no more than 20 nodes, and in order to handle larger problems, theyusually need to incorporate heuristics search (Xiang & Kim, 2013) or limit the maximum number of parents.
Revised: However, these exact algorithms usually work on small graphs efﬁciently (De Campos & Ji, 2011), and in order to handle larger problems with hundreds of variables, they need to incorporate heuristics search (Xiang & Kim, 2013) or limit the maximum number of parents.

Original: Recently, RL has been used to tackle several combinatorial problems such as maximum cut and the traveling salesman problem, due to their relatively simple reward mechanisms.
Revised: Recently, RL has been used to tackle several combinatorial problems such as maximum cut and the traveling salesman problem, due to their relatively simple reward mechanisms (Khalil et al., 2017).

Original: In combination with the encoder-decoder based pointer networks (Vinyals et al., 2015), Bello et al.
Revised: In combination with the encoder-decoder based pointer networks (Vinyals et al., 2015), Bello et al.

Original: (2016) showed thatan RLagent can have a better generalization even when the optimal solutions are used as labeled data in a supervised way.
Revised: (2016) showed that RL can have a better generalization even when the optimal solutions are used as labeled data in a supervised way.

Original: Kool et al.
Revised: Kool et al.

Original: (2019) further used an attention based encoder-decoder model for an improved performance.
Revised: (2019) further used an attention based encoder-decoder model for an improved performance.

Original: Khalil et al.

Original: (2017) did not use a separate encoder and decoder, but a single model exploiting graph structures.

Original: These works aim to learn a policy as a combinatorial solver based on the same structure of a particular type of combinatorial problems.
Revised: These works aim to learn a policy as a combinatorial solver based on the same structure of a particular type of combinatorial problems.

Original: However, various causal discovery tasks generally have different causal relationships, data types, graph structures, etc, and moreover, are typically off-line with focus on a causal graph.
Revised: However, various causal discovery tasks generally have different causal relationships, data types, graph structures, etc, and moreover, are typically off-line with focus on a causal graph.

Original: As such, we use RL as a search strategy, similar to Zhu et al.
Revised: As such, we use RL as a search strategy, similar to Zhu et al.

Original: (2020); Zoph & Le (2017); nevertheless, a pretrained model or a policy can offer a good starting point to speed up training, as shown in our evaluation results (Figure 3).
Revised: (2020); Zoph & Le (2017); nevertheless, a pretrained model or a policy can offer a good starting point to speed up training, as shown in our evaluation results (Figure 3).

Original:  B ACKGROUND 3. C AUSAL S TRUCTURE L EARNING
Revised:  B ACKGROUND 3. C AUSAL S TRUCTURE L EARNING

Original: Let G =
Revised: Let G =

Original: ( d, V, E ) denotes a DAG, with d the number of nodes, V = { v 1 , . .
Revised: ( d, V, E ) denotes a DAG, with d the number of nodes, V = { v 1 , . .

Original: , v d } the set of nodes, and E = { ( v i , v j ) | i, j = 1 , . . .
Revised: , v d } the set of nodes, and E = { ( v i , v j ) | i, j = 1 , . . .

Original: , d } the set of directed edges from v i to v j .
Revised: , d } the set of directed edges from v i to v j .

Original: Each node v j is associated with a random variable X j .
Revised: Each node v j is associated with a random variable X j .

Original: The probability model associated with G factorizes as p ( X 1 , . . .
Revised: The probability model associated with G factorizes as p ( X 1 , . . .

Original: , X d ) = (cid:81) dj
Revised: , X d ) = (cid:81) dj

Original: =1 p ( X j | Pa ( X j )) , where p ( X j |
Revised: =1 p ( X j | Pa ( X j )) , where p ( X j |

Original: Pa ( X j )) is the conditional probability distribution for X j given its parents Pa ( X j ) :
Revised: Pa ( X j )) is the conditional probability distribution for X j given its parents Pa ( X j ) :

Original: = { X k | ( v k , v j ) ∈ E } .
Revised: = { X k | ( v k , v j ) ∈ E } .

Original: We assume that the observed data x j is obtained by the
Revised: We assume that the observed data x j is obtained by the

Original: Structural Equation Model (SEM) with addtive noise: X j :=
Revised: Structural Equation Model (SEM) with addtive noise: X j :=

Original: f θ j ( Pa ( X j ))
Revised: f θ j ( Pa ( X j ))

Original: + (cid:15) j , j = 1 , . . .
Revised: + (cid:15) j , j = 1 , . . .

Original: , d , where f θ j parameterized by θ j is used to represent the functional relationship between X j and its parents, and (cid:15)
Revised: , d , where f θ j parameterized by θ j is used to represent the functional relationship between X j and its parents, and (cid:15)

Original: j ’s denote jointly independent additive noise variables.
Revised: j ’s denote jointly independent additive noise variables.

Original: We assume causal minimality, which is equivalent to that each f j is not a constant for any X k ∈ Pa ( X j ) in this SEM (Peters et al., 2014).
Revised: We assume causal minimality, which is equivalent to that each f j is not a constant for any X k ∈ Pa ( X j ) in this SEM (Peters et al., 2014).

Original: Given a sample X =
Revised: Given a sample X =

Original: [ x 1 , . . .
Revised: [ x 1 , . . .

Original: , x d ] , where X ∈ R m × d and x j is a vector of m observations for random variables X j .
Revised: , x d ] , where X ∈ R m × d and x j is a vector of m observations for random variables X j .

Original: Our goal in this paper is to ﬁnd the DAG G that maximizes the BIC score (or other well studied scores), deﬁned as
Revised: Our goal in this paper is to ﬁnd the DAG G that maximizes the BIC score (or other well studied scores), deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where | θ j | is the number of free parameters in p ( X kj | Pa ( X kj ) , θ j ) .
Revised: where | θ j | is the number of free parameters in p ( X kj | Pa ( X kj ) , θ j ) .

Original: For linear causal relationships, | θ j | = | Pa ( X kj ) | the number of parents, up to some constant factor.
Revised: For linear causal relationships, | θ j | = | Pa ( X kj ) | the number of parents, up to some constant factor.

Original: [Equation]
Revised: [Equation]

Original: The problem of ﬁnding a directed graph that satisﬁes the ayclicity constraint can be cast as that of ﬁnding an variable ordering (Teyssier & Koller, 2005;
Revised: The problem of ﬁnding a directed graph that satisﬁes the ayclicity constraint can be cast as that of ﬁnding an variable ordering (Teyssier & Koller, 2005;

Original: Schmidt et al., 2007).
Revised: Schmidt et al., 2007).

Revised: The score of an ordering is usually deﬁned as the score of the best DAG that is consistent with the given ordering.

Original: Speciﬁcally, let Π denote an ordering of the nodes in V , where the length of the ordering | Π | = | V | and Π is indexed from 1.
Revised: Speciﬁcally, let Π denote an ordering of the nodes in V , where the length of the ordering | Π | = | V | and Π is indexed from 1.

Original: If node v j ∈ V lies in the p -th position, then Π( p ) = v j .
Revised: If node v j ∈ V lies in the p -th position, then Π( p ) = v j .

Original: Notation Π ≺ v j denotes the set of nodes in V that precede node v j in Π .
Revised: Notation Π ≺ v j denotes the set of nodes in V that precede node v j in Π .

Original: One can establish a canonical correspondence between an ordering Π and a fully-connected DAG G Π ; an example with four nodes is presented in Figure 1.
Revised: One can establish a canonical correspondence between an ordering Π and a fully-connected DAG G Π ; an example with four nodes is presented in Figure 1.

Original: For a given DAG G , it can be consistent with more than one orderings and the set of these orderings is given by
Revised: For a given DAG G , it can be consistent with more than one orderings and the set of these orderings is given by

Original: [Equation]
Revised: [Equation]

Original: where a super-DAG of G is a DAG whose edge set is a superset of that of G .
Revised: where a super-DAG of G is a DAG whose edge set is a superset of that of G .

Original: The score of an ordering is usually deﬁned as the score of the best DAG that is consistent with the given ordering (Teyssier & Koller, 2005; Peters et al., 2014).
Revised: The score of an ordering is usually deﬁned as the score of the best DAG that is consistent with the given ordering (Teyssier & Koller, 2005; Peters et al., 2014).

Original: We provide a formal description in Proposition 1 to show that it is possible to ﬁnd the correct ordering with high probability in the large sample limit.
Revised: We provide a formal description in Proposition 1 to show that it is possible to ﬁnd the correct ordering with high probability in the large sample limit.

Original: Therefore, the search for the true DAG G ∗ can be decomposed to two phases: ﬁnding the correct ordering and performing variable selection (feature selection); the latter is to ﬁnd the optimal DAG that is consistent with the ordering found in the ﬁrst step.
Revised: Therefore, the search for the true DAG G ∗ can be decomposed to two phases: ﬁnding the correct ordering and performing variable selection (feature selection); the latter is to ﬁnd the optimal DAG that is consistent with the ordering found in the ﬁrst step.

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose that an identiﬁable SEM with true causal DAG G ∗ on X =
Revised: Suppose that an identiﬁable SEM with true causal DAG G ∗ on X =

Original: { X j } dj =1 induces distribution P ( X ) .
Revised: { X j } dj =1 induces distribution P ( X ) .

Original: Let G Π be the fully-connected DAG that corresponds to an ordering Π .
Revised: Let G Π be the fully-connected DAG that corresponds to an ordering Π .

Original: If there is an SEM with G Π inducing the same distribution P ( X ) , then G Π must be a super-graph of G ∗ , i.e., every edge in G ∗ is covered in G Π .
Revised: If there is an SEM with G Π inducing the same distribution P ( X ) , then G Π must be a super-graph of G ∗ , i.e., every edge in G ∗ is covered in G Π .

Original: Proof.
Revised: Proof.

Original: The SEM with G Π may not be causally minimal but can be reduced to an SEM satisfying the causal minimality condition (Peters et al., 2014).
Revised: The SEM with G Π may not be causally minimal but can be reduced to an SEM satisfying the causal minimality condition (Peters et al., 2014).

Original: Let ˜ G Π denotes the causal graph in the reduced
Revised: Let ˜ G Π denotes the causal graph in the reduced

Original: SEM with the same distribution P ( X ) .
Revised: SEM with the same distribution P ( X ) .

Original: Since we have assumed that original SEM is identiﬁable, i.e., the distribution P ( X ) corresponds to a unique true graph, ˜ G Π is then identical to G ∗ .
Revised: Since we have assumed that original SEM is identiﬁable, i.e., the distribution P ( X ) corresponds to a unique true graph, ˜ G Π is then identical to G ∗ .

Original: The proof is complete by noticing that G Π is a super-graph of ˜ G Π .
Revised: The proof is complete by noticing that G Π is a super-graph of ˜ G Π .

Original: 3. R EINFORCEMENT L EARNING
Revised: 3. R EINFORCEMENT L EARNING

Original: Standard RL is usually formulated as an MDP over the environment state s ∈ S and agent action a ∈ A , under an (unknown) environmental dynamics deﬁned by a transition probability T ( s (cid:48) | s, a ) .
Revised: Standard RL is usually formulated as an MDP over the environment state s ∈ S and agent action a ∈ A , under an (unknown) environmental dynamics deﬁned by a transition probability T ( s (cid:48) | s, a ) .

Original: We use π θ ( a | s ) to denote the policy, parameterized by θ , which outputs a discrete (or continuous) distribution used to select an action from action space
Revised: We use π φ ( a | s ) to denote the policy, parameterized by φ , which outputs a discrete (or continuous) distribution used to select an action from action space

Original: A based on state s .
Revised: A based on state s .

Original: For episodic tasks, a trajectory τ = { s t , a t } Tt =0 , where T is the ﬁnite time horizon, can be collected by executing the policy repeatedly.
Revised: For episodic tasks, a trajectory τ = { s t , a t } Tt =0 , where T is the ﬁnite time horizon, can be collected by executing the policy repeatedly.

Original: In many cases, an immediate reward r ( s, a ) can be received when agent executes an action.
Revised: In many cases, an immediate reward r ( s, a ) can be received when agent executes an action.

Original: The objective of RL is to learn a policy which can maximize the expected cumulative reward along a trajectory, i.e., J ( θ ) =
Revised: The objective of RL is to learn a policy which can maximize the expected cumulative reward along a trajectory, i.e., J ( φ ) =

Original: E π θ [ R 0 ] with R 0 = (cid:80) Tt =

Revised: E π φ [ R 0 ] with R 0 = (cid:80) Tt =

Original: γ t r t ( s t , a t ) and γ ∈ (0 , 1] being a discount factor.
Revised: γ t r t ( s t , a t ) and γ ∈ (0 , 1] being a discount factor.

Original: For some scenarios, the reward is only earned at the terminal time (also called episodic reward), and J ( θ ) =
Revised: For some scenarios, the reward is only earned at the terminal time (also called episodic reward), and J ( φ ) =

Original: E π θ [ R ( τ )] with R ( τ ) =
Revised: E π φ [ R ( τ )] with R ( τ ) =

Original: r T ( s T , a T ) .
Revised: r T ( s T , a T ) .

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we ﬁrst introduce how to model the ordering search problem as an MDP, then we show how to use RL to ﬁnd the optimal ordering efﬁciently, andﬁnally we introduce how to process the searched ordering to obtain the ﬁnal DAG.
Revised: In this section, we ﬁrst introduce how to model the ordering search problem as an MDP, then we show how to use RL to ﬁnd the optimal ordering, and we introduce how to process the searched ordering to obtain the ﬁnal DAG, ﬁnally we provide a discussion on computational complexity.

Original: 4. O RDERING S EARCH AS A M ARKOV D ECISION P ROCESS
Revised: 4. O RDERING S EARCH AS A M ARKOV D ECISION P ROCESS

Original: We can regard the variable ordering search problem as a muti-stage decision problem with a variable selected at each decision step.
Revised: We can regard the variable ordering search problem as a muti-stage decision problem with a variable selected at each decision step.

Original: We sort the selected variables according to decision steps to obtain a variable ordering, which is deﬁned as the searched ordering.
Revised: We sort the selected variables according to decision steps to obtain a variable ordering, which is deﬁned as the searched ordering.

Original: The decision-making process is
Revised: The decision-making process is

Original: Markovian, and its elements in the problem can be deﬁned as follows:
Revised: Markovian, and its elements in the problem can be deﬁned as follows:

Original:  One can directly take the sample data x j of each variable X j as a state s j .
Revised: State One can directly take the sample data x j of each variable X j as a state s j .

Original: However, previous experiments show that it is difﬁcult for feed-forward neural network models to capture the underlying causal relationships directly using observed data as states. Therefore we follow Zhu et al. andadopt the attention based encoder in the Transformer structure proposed by Vaswani et al.
Revised: However , preliminary experiments show that it is difﬁcult for feed-forward neural network models to capture the underlying causal relationships directly using observed data as states, and the data pre-processed by a module conventionally named encoder is helpful to ﬁnding the better ordering, see Appendix A.1.

Original: The encoder module embeds each x j to s j and all the embedded states constitute thestate space S :
Revised: The encoder module embeds each x j to s j and all the embedded states constitute the space S :

Original: = { s 1 , . . .
Revised: = { s 1 , . . .

Original: , s d } .
Revised: , s d } .

Original: We pick the initial state[-as s 
 = 1 d (cid:80) di =
-] s i and denote ˆ S :
Revised: Adding initial state[-as s 
 = 1 d (cid:80) di =
-] s 0 to the space constitutes the complete state space ˆ S :

Original: = S ∪ s [-
-]
Revised: = S ∪ s [-
-]0 .

Original: the new state space.

Original: We use ˆ s t to represent the state encountered at the t -th decision step.
Revised: We use ˆ s t to denote the state encountered at the t -th decision step.

Original: Action
Revised: Action

Original: We select an action (variable) from the action space constituted by all the variables at each decision step, and the space size is equal to the number of variables, |A| =
Revised: We select an action (variable) from the action space constituted by all the variables at each decision step, and the space size is equal to the number of variables, |A| =

Original: d .
Revised: d .

Original: Compared to the previous RL-based method searching from the graph space with size O (2 d × d ) (Zhu et al., 2020), the search space is smaller.
Revised: Compared to the previous RL-based method searching from the graph space with size O (2 d × d ) (Zhu et al., 2020), the search space is smaller.

Original: Note that according to the deﬁnition of ordering, the ﬁrst selected variable is the source node and the last selected node is the sink node.
Revised: Note that according to the deﬁnition of ordering, the ﬁrst selected variable is the source node and the last selected node is the sink node.

Original: State transition
Revised: State transition

Original: The speciﬁed state transition is related to the action selected at the current decision step.
Revised: The speciﬁed state transition is related to the action selected at the current decision step.

Original: Speciﬁcally, if the selected variable is v j at the t -th decision step, then the state is transferred to the state s j ∈ S which is the j -th output from Transformer encoder, i.e., ˆ s t +1 =
Revised: Speciﬁcally, if the selected variable is v j at the t -th decision step, then the state is transferred to the state s j ∈ S which is the j -th output from Transformer encoder, i.e., ˆ s t +1 =

Original: s j .
Revised: s j .

Original: Reward
Revised: Reward

Original: As we described in Section 3.1, only the variables that have been selected in previous decision steps can be the potential parents of the currently selected variable.
Revised: As we described in Section 3.1, only the variables that have been selected in previous decision steps can be the potential parents of the currently selected variable.

Original: Hence, we can design the rewards in the following cases: dense reward and episodic reward .
Revised: Hence, we can design the rewards in the following cases: dense reward and episodic reward .

Original: For dense reward case, we can exploit the decomposability of the score function (BIC score) to calculate an immediate reward for the current decision step based on the potential parent variables (that have been selected), i.e., if v j is selected at time step t , the immediate reward is calculated by
Revised: For dense reward case, we can exploit the decomposability of the score function (BIC score) to calculate an immediate reward for the current decision step based on the potential parent variables (that have been selected), i.e., if v j is selected at time step t , the immediate reward is calculated by

Original: [Equation]
Revised: [Equation]

Original: where U ( X j ) denotes the potential parent variable set of X j and the set consists of the variables associated with the nodes in Π ≺ v j .
Revised: where U ( X j ) denotes the potential parent variable set of X j and the set consists of the variables associated with the nodes in Π ≺ v j .

Original: For episodic reward case, we directly calculate a score as an episodic reward for a complete variable ordering regardless of whether the scoring function is decomposable or not, i.e.,
Revised: For episodic reward case, we directly calculate a score as an episodic reward for a complete variable ordering regardless of whether the scoring function is decomposable or not, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where Score BIC has been deﬁned in Equation (1), with the set Pa ( X j ) of each variable X j replaced by the potential parent variable set U ( X j ) here.
Revised: where Score BIC has been deﬁned in Equation (1), with the set Pa ( X j ) of each variable X j replaced by the potential parent variable set U ( X j ) here.

Original: 4. I MPLEMENTATION AND O PTIMIZATION WITH R EINFORCEMENT L EARNING
Revised: 4. I MPLEMENTATION AND O PTIMIZATION WITH R EINFORCEMENT L EARNING

Original: We describe the neural network architectures implemented in our policy model, which consists of an encoder and a decoder as shown in Figure 2.
Revised: We describe the neural network architectures implemented in our method, which consists of an encoder and a decoder as shown in Figure 2.

Original: Here we brieﬂy describe the model architectures and leave details regarding model parameters to Appendix A.
Revised: Here we brieﬂy describe the model architectures and leave details regarding model parameters to Appendix A.

Original: [Figure]
Revised: [Figure]

Original:  f enc θ e :  X (cid:55)→ S maps X =
Revised: Encoder f enc φ e : ˜ X (cid:55)→ S is used to map the observed data to the embedding space S

Original: [ x 1 , . . .
Revised: = { s 1 , . . .

Original: , x d ] to the space S = { s 1 , . . .

Original: , s d } .
Revised: , s d } .

Original: As previously explained in Section 4.1, we  use  the attention based encoder in the Transformer structure (Vaswani et al., 2017) as the state processing module.
Revised: For sample efﬁciency, we follow Zhu et al. (2020) to randomly draw n samples from the dataset X to construct ˜ X ∈ R n × d at each episode and use ˜ X instead of X . We also set the embedding s j to be in the same dimension, i.e., s j ∈ R n .

Original: Other attention based models such as graph attention network (Veliˇckovi´c et al., 2018) may also be adopted in our model.
Revised: For encoder choice, we conduct an empirical comparison among several possible structures such as a selfattention based encoder (Vaswani et al., 2017) and an LSTM structure. Empirically, we conﬁrm that the self-attention based encoder in the Transformer structure performs the best, which is also used in Zhu et al. Please ﬁnd more details in Appendix A.1.

Original: Decoder f [Equation]
Revised: Decoder f dec φ d : ˆ S (cid:55)→

Original: A maps the state space ˆ S to the action space A .
Revised: A maps the state space ˆ S to the action space A .

Original: For the multi-stage decision task, we implement the decoder module using a recurrent neural network, speciﬁcally  LSTM neural network.{+based structure that proves effective in our experiments. Although the initial state is usually generated randomly, we pick it as s 
 = 1 d (cid:80) di =1 s i , considering that the source node is ﬁxed in correct ordering.+}
Revised: Among several decoder choices (please see also Appendix A.1 for empirical comparison), we speciﬁcally pick an LSTM {+based structure that proves effective in our experiments. Although the initial state is usually generated randomly, we pick it as s 
 = 1 d (cid:80) di =1 s i , considering that the source node is ﬁxed in correct ordering.+}

Original: We  restrictthat each node only be selected once by masking the selected nodes so as to generate a valid variable ordering (Vinyals et al., 2015).
Revised: We then restrict each node only be selected once by masking the selected nodes so as to generate a valid variable ordering (Vinyals et al., 2015).

Original: The optimization objective is to learn a policy which maximizes J ( θ ) =
Revised: The optimization objective is to learn a policy which maximizes J ( φ ) =

Original: E π θ [ R ] , where π θ denotes the policy model parameterized by the paprameters { θ e , θ d } of encoder f enc and decoder f dec .
Revised: E π φ [ R ] , where π φ denotes the policy model parameterized by the paprameters { φ e , φ d } of encoder f enc and decoder f dec .

Original: Based on the above deﬁnition, policy gradient (Sutton & Barto, 2018) is used to optimize the ordering generation model parameters.
Revised: Based on the above deﬁnition, policy gradient (Sutton & Barto, 2018) is used to optimize the ordering generation model parameters.

Original: For the dense reward case, policy gradient can be written as ∇ J ( θ ) =
Revised: For the dense reward case, policy gradient can be written as ∇ J ( φ ) =

Original: E π θ (cid:104)(cid:80) Tt =

Revised: E π φ (cid:104)(cid:80) Tt =

Original: R t ∇ θ log π θ ( a t | ˆ s t ) (cid:105) , where R t = (cid:80) T − t l =

Revised: R t ∇ φ log π φ ( a t | ˆ s t ) (cid:105) , where R t = (cid:80) T − t l =

Original: γ l r t + l denotes the return at time step t .
Revised: γ l r t + l denotes the return at time step t .

Original: We denote the algorithm using this reward design as CORL-1.
Revised: We denote the algorithm using this reward design as CORL-1.

Original: For the episodic reward case, we have the following policy gradient ∇ J ( θ ) =
Revised: For the episodic reward case, we have the following policy gradient ∇ J ( φ ) =

Original: E π θ (cid:104) R ( τ ) (cid:80) Tt =0 ∇ θ log π θ ( a t | ˆ s t ) (cid:105) , the algorithm using this reward design is denoted as CORL-2.
Revised: E π φ (cid:104) R ( τ ) (cid:80) Tt =0 ∇ φ log π φ ( a t | ˆ s t ) (cid:105) , the algorithm using this reward design is denoted as CORL-2.

Original: Using a parametric baseline to estimate the expected score typically improves learning (Konda & Tsitsiklis, 2000; Zhu et al., 2020).
Revised: Using a parametric baseline to estimate the expected score typically improves learning (Konda & Tsitsiklis, 2000).

Original: Therefore, we introduce a critic network V θ v (ˆ s t ) parameterized by θ v , which learns the expected return given a state ˆ s t .
Revised: Therefore, we introduce a critic network V φ v (ˆ s t ) parameterized by φ v , which learns the expected return given a state ˆ s t .

Original: It is trained with stochastic gradient descent using Adam optimizer on a mean squared error objective between its predicted value and the actual return.
Revised: It is trained with stochastic gradient descent using Adam optimizer on a mean squared error objective between its predicted value and the actual return.

Original: For details about the parameters of critic, see also Appendix A.
Revised: The details about the parameters of critic network are described in Appendix A.2.

Original: Inspired by the beneﬁts of pretrained models in other tasks (Hinton & Salakhutdinov, 2012), we also consider to incorporate it to our method to accelerate training.
Revised: Inspired by the beneﬁts of pretrained models in other tasks (Hinton & Salakhutdinov, 2012), we also consider to incorporate it to our method to accelerate training.

Original: Usually, one can obtain some observed data with known causal structure or correct ordering, e.g., by simulation or real data with labeled graph.
Revised: Usually, one can obtain some observed data with known causal structure or correct ordering, e.g., by simulation or real data with labeled graph.

Original: Hence, weconsider to pretrain a policy model with such data in a supervised way.
Revised: Hence, we pretrain a policy model with such data in a supervised way.

Original: To learn a general policy instead of overﬁtting to the data pattern, we use the observed data from different types of  causal relationships, different sizes of graphs and different noise types to pretrain a policy model.
Revised: So far we presented CORL in a general manner without specifying explicitly which distribution family is during the evaluation of rewards. In principle, any distribution family could be employed as long as its log-likelihood can be computed and no differentiability is required. However, it is not always clear whether the maximization of the accumulated reward recovers the correct ordering. It will depend on both the modelling choice of reward and the underlying SEM; in fact, if the causal relationships fall into the chosen model functions and a right distribution family is assumed, then given inﬁnite samples the optimal accumulated reward, corresponding to the negative log-likelihood, must be achieved by a super-DAG of the underlying graph according to Proposition 1.

Original: To demonstrate the generalization ability of the RL method, we also consider a scenario where the policy model learned on a data model task is used as the pretrained policy model  for other tasks.
Revised: In practice, we Algorithm 1 CORL. 11: update 13: end while 14: get the ﬁnal DAG by pruning the BestOrdering can only apply approximate model functions and also need to assume certain distribution family for caculating the reward.

Original: Such results are demonstrated in Section 5.1.
Revised: Our method is summarized in Algorithm 1.

Original: In addition, we record the decomposed scores for each variable v j with different parental sets Π ≺ v j to avoid repeated computations which are generally time-consuming.
Revised: In addition, we record the decomposed scores for each variable v j with different parental sets Π ≺ v j to avoid repeated computations which are generally time-consuming (see Section 4.4).

Original: We denote v t as the node that is associated with the action a t . Although  policy gradient methods only guarantee local convergenceunder proper conditions (Sutton & Barto, 2018), we remark that the orderings obtained from the actor-critic algorithm are promising in our experiments. Our algorithm is summarized in Algorithm 1.
Revised: Although we cannot guarantee to ﬁnd the optimal ordering because policy gradient can at most guarantee local convergence (Sutton et al., 2000) and also we only have access to the empirical log-likelihood, we remark that the ordering obtained from CORL still enjoy a good performance in the experiments, compared with consistent methods like GES and PC.

Original: 4. V ARIABLE S ELECTION
Revised: 4. V ARIABLE S ELECTION

Original: If an estimated ordering ˆΠ is consistent, then we obtain a fully-connected DAG (super-DAG)
Revised: If an estimated ordering ˆΠ is consistent, then we obtain a fully-connected DAG (super-DAG)

Original: G ˆΠ of the underlying DAG G .
Revised: G ˆΠ of the underlying DAG G .

Original: One can then pursue consistent estimation of intervention distributions based on ˆΠ without any additional need to ﬁnd the true underlying DAG G ∗ (Bühlmann et al., 2014).
Revised: One can then pursue consistent estimation of intervention distributions based on ˆΠ without any additional need to ﬁnd the true underlying DAG G ∗ (Bühlmann et al., 2014).

Original: For other purposes, however, we need to recover the true graph from the fully-connected DAG.There
Revised: For other purposes, however, we need to recover the true graph from the fully-connected DAG.

Original: Algorithm 1 CORL. 11: update BestOrdering 12: end 13: end while 14: get the ﬁnal DAG by pruning the BestOrdering exist several efﬁcient methods such as sparse candidate (Teyssier & Koller, 2005), signiﬁcance testing of covariates (Bühlmann et al., 2014), the group Lasso (Ravikumar et al., 2009), or its improved version with a sparsity-smoothness penalty proposed in Meier et al.
Revised: There exist several efﬁcient methods such as sparse candidate (Teyssier & Koller, 2005), signiﬁcance testing of covariates (Bühlmann et al., 2014), the group Lasso (Ravikumar et al., 2009), or its improved version with a sparsity-smoothness penalty proposed in Meier et al.

Original: For linear data models, we apply linear regression to the obtained fully-connected DAG, followed by tresholding to prune edges with small weights.
Revised: For linear data models, we apply linear regression to the obtained fully-connected DAG, followed by thresholding to prune edges with small weights, as similarly used by Zheng et al.

Original: This method is particularly useful in the case of obtaining the coefﬁcient range of the causal functional model, see Section 5.1.
Revised: Yu et al. (2019); Zhu et al.

Original: For the nonlinear model, we follow the pruning process used by Bühlmann et al.
Revised: For the nonlinear model, we follow the pruning process used by Bühlmann et al.

Original: ; Lachapelle et al.
Revised: ; Lachapelle et al.

Original: Speciﬁcally, for each variable X j ,  a generalized additive modelis ﬁtted against the current parents of X j and a signiﬁcance test of covariates is applied.
Revised: Speciﬁcally, for each variable X j , one can ﬁt a generalized additive model against the current parents of X j and then apply signiﬁcance testing of covariates, declaring signiﬁcance if the reported p-values are lower or equal to 0 . 4. C OMPUTATIONAL C OMPLEXITY

Revised: To learn an ordering, CORL relies on the proper training of RL model.

Original: Nodes with  a p -value higher than 0 .
Revised: Policy gradient and stochastic gradient are adopted to train the actor and critic respectively, which are the standard choice in RL (Konda & Tsitsiklis, 2000). Similar to RL-BIC2 (Zhu et al., 2020), CORL requires the evaluation of the rewards at each episode with O ( dm 2 + d 3 ) computational cost if linear functions are adopted to model the causal relations, but does not need to compute the matrix exponential term with O ( d 3 ) cost due to the use of ordering search. In addition, CORL formulates causal discovery as a multi-stage decision process and we observe that CORL performs fewer episodes than RL-BIC2 before the episode reward converges (see Appendix C). We suspect that due to a signiﬁcant reduction in the size of action space, the model complexity of the RL policy is reduced, thus leading to higher sample efﬁciency.

Revised: The evaluation of Transformer encoder and LSTM decoder in CORL take O ( nd 2 ) and

Revised: O ( dn 2 ) , respectively.

Original: 001 are removed from the parent set.
Revised: However, we ﬁnd that computing rewards is more dominating in the total runing time (e.g., around 95% and 87% for 30- and 100-node linear data models, respectively). Speeding up the calculation of rewards would be helpful in extend our approach to a larger problem, which is left as a future work. In contrast with typical RL applications, we treat RL here as a search strategy for causal discovery, aiming to ﬁnd an ordering that achieves the best score and then applying a variable selection method to remove redundant edges. Nevertheless, for the pretraining part with the goal of a good initialization, we may want sufﬁcient generalization ability and hence consider diverse datasets with different number of nodes, noise types, causal relationships, etc.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate our methods against a number of methods on synthetic datasets with linear and non-linear causal relationships and also a real dataset.
Revised: In this section, we evaluate our methods against a number of methods on synthetic datasets with linear and non-linear causal relationships and also a real dataset.

Original: Speciﬁcally, these baselines include
Revised: Speciﬁcally, these baselines include

Original: ICA-LiNGAM (Shimizu et al., 2006), three ordering-based approaches L1OBS (Schmidt et al., 2007),
Revised: ICA-LiNGAM (Shimizu et al., 2006), three ordering-based approaches L1OBS (Schmidt et al., 2007),

Original: CAM (Bühlmann et al., 2014) and A* Lasso (Xiang & Kim, 2013), some recent gradient-based approaches NOTEARS (Zheng et al., 2018), DAG-GNN (Yu et al., 2019) and GraN-DAG (Lachapelle et al., 2020), and the RL-based approach RL-BIC2 (Zhu et al., 2020).
Revised: CAM (Bühlmann et al., 2014) and A* Lasso (Xiang & Kim, 2013), some recent gradient-based approaches NOTEARS (Zheng et al., 2018), DAG-GNN (Yu et al., 2019) and GraN-DAG (Lachapelle et al., 2020), and the RL-based approach RL-BIC2 (Zhu et al., 2020).

Original: For all the compared algorithms, we use their original implementations  and pick the recommended hyper-parameters unless otherwise stated.
Revised: For all the compared algorithms, we use their original implementations (see Appendix B.1 for details) and pick the recommended hyper-parameters unless otherwise stated.

Original: More details are provided in Appendix A.
Revised: Different types of data are generated in synthetic datasets which vary along ﬁve dimensions: level of edge sparsity, graph type, number of nodes, causal functions and sample size.

Original: The observed data can be obtained by exploiting the DAG G drawn randomly from the Erdös–Rényi (ER) graph model.
Revised: Two types of graph sampling schemes: Erdös–Rényi (ER) and Scale-free (SF) are considered in our experiments.

Original: We denote d -node ER  graphs with on average nd edges as ER n .
Revised: We denote d -node ER or SF graphs with on average hd edges as ER h or SF h .

Original: Two common metrics are considered:1) True Positive Rate (TPR) which indicates the probability of correctly ﬁnding the positive edges among the discoveries (Jain et al., 2017).Structural Hamming
Revised: Two common metrics are considered: True Positive Rate (TPR) and Structural Hamming Distance (SHD). The former indicates the probability of correctly ﬁnding the positive edges among the discoveries (Jain et al., 2017).

Original: Distance (SHD) which indicates the total number of edge additions, deletions, and reversals needed to convert the estimated graph into the true DAG. For TPR, the higher the better, while for SHD, the smaller the better.
Revised: Hence, it can be used to measure the quality of an ordering, the higher the better. The latter counts the total number of missing, falsely detected or reversed edges, the smaller the better.

Original: 5. L INEAR D ATA M ODELS WITH G AUSSIAN AND N ON -G AUSSIAN N OISE
Revised: 5. L INEAR D ATA M ODELS WITH G AUSSIAN AND N ON -G AUSSIAN N OISE

Original: We evaluate the proposed methods on graphs with linear Gaussian  and LiNGAM data models.
Revised: We evaluate the proposed methods on Linear Gaussian (LG) with equal variance Gaussian noise and LiNGAM data models, and the true DAGs in both cases are known to be identiﬁable (Peters & Bühlmann, 2014; Shimizu et al., 2006).

Original: We set n = 2 . 5  and follow Zheng et al.Zhu et al.
Revised: We set h ∈ { 2 , 5 } and d ∈ { 30 , 50 , 100 } to generate observed data following the procedure done in Zheng et al.

Original: (2020) to generate m = 3000 observed data with d ∈ { 30 , 50 , 100 } nodes according to X = W T X + (cid:15) , where W ∈ R d × d denotes the weight matrix which is obtained by assigning edge weights independently from Unif ([ − 2 , − 0 . 5] ∪ [0 . 5 , 2]) and (cid:15) ’s are standard Gaussian noises with linear Gaussian data models.
Revised: (see Appendix B.2 for details).

Original: For variable selection, we set the thresholding as 0.3 and apply it to the estimated coefﬁcients.
Revised: For variable selection, we set the thresholding as 0.3 and apply it to the estimated coefﬁcients.

Original: [Table]
Revised: [Table]

Original:  We report the empirical results on  the linear Gaussian datasets in Table 1.
Revised: Tables 1 &2 present results only for 30 - and 100 -node LG data models since the conclusions do not change with graphs of 50 nodes (see Appendix D for 50-node graphs). We report the performance of the popular ICA-LiNGAM, GraN-DAG and CAM in Appendix D since they are almost never on par with the best methods presented in this section.

Original: CORL-1 and CORLachieve consistently good results on the LiNGAM datasets which are reported in Appendix B due to the space limit.
Revised: CORL-1 and CORL-2 achieve consistently good results on the LiNGAM datasets which are reported in Appendix E due to the space limit.

Original: It is not surprising that ICA-LiNGAM does not perform well with linear Gaussian data models, because the algorithm is speciﬁcally designed for non-Gaussian noise and does not provide guarantee for linear Gaussian data models.
Revised: We now examine Tables 1 &2 (the values in parentheses represent the standard deviation across datasets per task).

Original: Both CAM and GraN-DAG perform poorly on the linear datasets. We believe that this is because CAM uses nonlinear regression instead of linear regression.
Revised: We can see that, across all settings, CORL-1 and CORL-2 are the best performing methods, both in terms of TPR and SHD, while NOTEARS and DAG-GNN are not too far behind.

Original: As for GraN-DAG, it uses 2 -layer feed-forward neural networks to model the causal relationships, which may not be able to learn a good linear relationship in this experiment.
Revised: [Table]

Original: As we discussed previously, RL-BIC2 only achieves satisfactory results on graphs with 30 nodes.
Revised: As we discussed previously, RL-BIC2 only achieves satisfactory results on graphs with 30 nodes.

Original: Both NOTEARS and DAG-GNN have good causal discovery results on all the linear datasets, while

Original: L1OBS and A* Lasso achieve worse performance than NOTEARS and DAG-GNN.

Original: The TPR of L1OBS is lower than that of A* Lasso, which indicates that L1OBS using greedy hill-climbing with tabu lists may not ﬁnd a good ordering.
Revised: The TPR of L1OBS is lower than that of A* Lasso, which indicates that L1OBS using greedy hill-climbing with tabu lists may not ﬁnd a good ordering.

Original: We notice that SHD of L1OBS and A* Lasso is even worse than that of CAM, which may be due to the different variable selection methods. Applying our pruning method to L1OBS and A* Lasso, we observe that SHD of them is greatly improved.
Revised: Note that SHD of L1OBS and A* Lasso reported here are the results of applying our pruning method. We observe that SHD of them is greatly improved by pruning.

Original: For example,  SHD of L1OBS decreases from 171 .
Revised: For example, speciﬁcally, SHD of L1OBS decreases from 171 .

Original: 6 (29 . 5) , 588 . 0 (66 . 2) and 1964 .
Revised: 2) and 1964 .

Original: 5 (136 . 6) to 85.(23 .
Revised: 5 (136 . 6) to 85 .

Original: (26 . 3) and 481 .
Revised: (26 . 3) and 481 .

Original: 2 (49 . 9) in different data models (linear Gaussian with 30 , 50 andnodes, respectively) while TPR almost did not degraded.
Revised: 2 (49 . 9) in LG ER2 with 30 , 50 and 100 nodes respectively, while TPR almost did not degraded.

Original: It shows the effectiveness of pruning by model ﬁtting with a thresholding; see Appendix B for the pruned results.
Revised: It shows the effectiveness of pruning by model ﬁtting with a thresholding.

Original: We further evaluate our method on 150 -node linear Gaussian data models.
Revised: We further evaluate our method on 150 -node LG ER2 data models.

Original: On the datasets, CORLhas TPR and SHD being 0 .
Revised: On the datasets, CORL-1 has TPR and SHD being 0 .

Original: 95 (0 . 01) and 63 .
Revised: 95 (0 . 01) and 63 .

Original: 7 (9 . 1) , while CORL-2 has 0 .
Revised: 7 (9 . 1) , while CORL-2 has 0 .

Original: 97 (0 . 01) and 38 .
Revised: 97 (0 . 01) and 38 .

Original: (14 . 3) , respectively.
Revised: (14 . 3) , respectively.

Original: CORL-2 outperforms NOTEARS with 0 .
Revised: CORL-2 outperforms NOTEARS with 0 .

Original: 94 (0 . 02) and 50 . 8 (21 .
Revised: 94 (0 . 02) and 50 . 8 (21 .

Original: [Figure]
Revised: [Figure]

Original: Although both CORL-1 and CORL-2 achieve the desired performance, one can notice that CORL-2 achieves slightly better SHD than CORL-1.
Revised: Although both CORL-1 and CORL-2 achieve the desired performance, one can notice that CORL-2 achieves slightly better SHD than CORL-1.

Original: This is a little different from the usual understanding that RL is usually easier to learn from dense rewards than from episodic reward case.
Revised: This is a little different from the usual understanding that RL is usually easier to learn from dense rewards than from episodic reward case.

Original: We demonstrate more details about our method.

Original: We take the 100-node linear Gaussian data models as an example to show the training reward curves of the CORLand CORL-2 in Figure 3.
Revised: We take the 100-node LG ER2 data models as an example to show the training reward curves of the CORLand CORL-2 in Figure 3.

Original: One can notice that CORLconverges faster to a better result than CORL-1, which corresponds to the fact that CORL-2 achieves the slightly better TPR and SHD than CORL-1.
Revised: One can notice that CORLconverges faster to a better result than CORL-1, which corresponds to the fact that CORL-2 achieves the slightly better TPR and SHD than CORL-1.

Original: We conjecture that this is because it is difﬁcult for the critic to learn to predict the score accurately for each state; in the episodic reward case, however, it is only required to learn to predict the score accurately for the initial state.
Revised: We conjecture that this is because it is difﬁcult for the critic to learn to predict the score accurately for each state; in the episodic reward case, however, it is only required to learn to predict the score accurately for the initial state.

Original: Wealso show the training reward curve of CORL-2-pretrain, which is CORL-2 based on the pretrained model trained in the supervised way.
Revised: We show the training reward curve of CORL-2-pretrain in Figure 3, which is CORL-2 based on a pretrained model trained in the supervised way.

Revised: The testing data task is unseen in the datasets for pretraining which contain 30-node

Original:  Obviously compared to that of CORL-2 using random initialization (CORL-2), the use of a pretrained model can accelerate the CORL-2 learning process. We also incorporate a pretrained model in CORL-1 and achieve a similar acceleration effect.
Revised: LiNGAM ER2, 50-node LiNGAM ER2, 30-node LiNGAM SF2 and 30-node GP ER1. Obviously compared to that of CORL-2 using random initialization (CORL-2), the use of a pretrained model can accelerate the model learning process. Consistent conclusion can be drawn from the experiment of

Original: We report the training reward curve in Appendix D.
Revised: CORL-1, see Appendix G.

Original: In addition, we consider using the policy model learned  on 30-node LiNGAM data models by RL as the pretrained model onthe 100-node graph tasks.
Revised: In addition, we consider using the policy model learned only on 30-node LiNGAM data model as the pretrained model on 100-node LG task.

Original: We observe that the performance is similar to that ofpolicy model using the pretrained model obtained in the supervised way, we do not report the results repeatedly here.
Revised: We observe that the performance is similar to that of using the pretrained model obtained in the supervised way, we do not report the results repeatedly here.

Original: 5. N ON -L INEAR M ODEL WITH G AUSSIAN P ROCESS
Revised: 5. N ON -L INEAR M ODEL WITH G AUSSIAN P ROCESS

Original: [Figure] In this experiment, we consider to use Gaussian Process (GP) to model causal relationships in which each causal relation f j is a function sampled from a GP with radial basis function kernel of bandwidth one and normal Gaussian noises, following the setting in Peters et al.(2014); Zhu et al.
Revised: In this experiment, we consider to use Gaussian Process (GP) to model causal relationships in which each causal relation f j is a function sampled from a GP with radial basis function kernel of bandwidth one and normal Gaussian noises, which is known to be identiﬁable according to Peters et al.

Original: We set n = 1 and 4 to get ER1 and ER4 graphs, respectively, and generate data by exploiting them.
Revised: We set h = 1 and 4 to get ER1 and ER4 graphs, respectively, and generate data by exploiting them (see Appendix B.2 for details).

Original: Our method is evaluated on different sample numbers, the results when m = 500 are reported here and other results can be found in Appendix C.
Revised: Our method is evaluated on different sample numbers, the results when m = 500 are reported here (see Appendix F for additional results). Note that due to the efﬁciency of [Figure]

Revised: the reward calculation, we only experimented on nonlinear data models of up to 30 nodes scale.

Original: The pruning method for variable selection used here is the CAM pruning from Bühlmann et al.
Revised: The pruning method for variable selection used here is the CAM pruning from Bühlmann et al.

Original: The results on 10-node and 30-node datasets with ER1 and ER4 graphs are shown in Figure 3.
Revised: The results on 10-node and 30-node datasets with ER1 and ER4 graphs are shown in Figure 3.

Original: Here we only consider some baselines that are competitive in the nonlinear data models, where CAM is a very strong ordering based baseline.
Revised: Here we only consider some baselines that are competitive in the nonlinear data models, where CAM is a very strong ordering based baseline.

Original: Although GraN-DAG achieves better results than DAG-GNN, they are worse than CAM overall.
Revised: Although GraN-DAG achieves better results than DAG-GNN, they are worse than CAM overall.

Original: We believe this is because 500 samples are so small that GraNDAG and DAG-GNN have not learned the good model.
Revised: We believe this is because 500 samples are so small that GraN-DAG and DAG-GNN have not learned the good model.

Original: RL-BIC2 performs well on 10-node datasets, but achieves poor results on 30-node datasets, probably due to its lack of scalability.
Revised: RL-BIC2 performs well on 10-node datasets, but achieves poor results on 30-node datasets, probably due to its lack of scalability.

Original: CAM, CORLand CORL-2 have good results, with CORL-2 achieving the best results onthe 10-node graphs and slightly worse than CAM onthe 30-node graph.
Revised: CAM, CORL-1 and CORL-2 have good results, with CORL-2 achieving the best results on 10-node graphs and slightly worse than CAM on 30-node graphs.

Original: All of these methods have better results on ER1 than on ER4, especially on 30-node graphs.
Revised: All of these methods have better results on ER1 than on ER4, especially on 30-node graphs.

Original: 5. R EAL D ATA
Revised: 5. R EAL D ATA

Original: The Sachs dataset (Sachs et al., 2005), with 11 -node and 17 -edge true graph, is widely used for research on graphical models.
Revised: The Sachs dataset (Sachs et al., 2005), with 11 -node and 17 -edge true graph, is widely used for research on graphical models.

Original: The expression levels of protein and phospholipid in the dataset can be used to discover the implicit protein signal network.
Revised: The expression levels of protein and phospholipid in the dataset can be used to discover the implicit protein signal network.

Original: The observational dataset has m = 853 samples and is used to discover the causal structure.
Revised: The observational dataset has m = 853 samples and is used to discover the causal structure.

Original: GP is used to model the causal relationship in our method.
Revised: GP is used to model the causal relationship in our method.

Original: In this experiment, CORL-1, CORL-2 and RL-BIC2 achieve the best SHD 11 .
Revised: In this experiment, CORL-1, CORL-2 and RL-BIC2 achieve the best SHD 11 .

Original: CAM, GraN-DAG, and
Revised: CAM, GraN-DAG, and

Original: ICA-LiNGAM achieve the SHD 12 , 13 and 14 , respectively.
Revised: ICA-LiNGAM achieve the SHD 12 , 13 and 14 , respectively.

Original: Particularly, DAG-GNN and NOTEARS result in SHD 16 and 19 , respectively, whereas an empty graph has an SHD 17 .
Revised: Particularly, DAG-GNN and NOTEARS result in SHD 16 and 19 , respectively, whereas an empty graph has an SHD 17 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have proposed a RL-based approach for causal discovery named CORL.
Revised: In this work, we have proposed a RL-based approach for causal discovery named CORL.

Original: It searches the space of variable orderings, instead of the space of directed graphs.
Revised: It searches the space of variable orderings, instead of the space of directed graphs.

Original: We formulate ordering search as an MDP and have further proposed CORL-1 and CORL-2 for training the ordering generating model.
Revised: We formulate ordering search as an MDP and have further proposed CORL-1 and CORL-2 for training the ordering generating model.

Original: For a generated ordering, it can be pruned by variable selection to the ﬁnal DAG.
Revised: For a generated ordering, it can be pruned by variable selection to the ﬁnal DAG.

Original: The empirical results on the synthetic and the real datasets show that our approach is promising.
Revised: The empirical results on the synthetic and the real datasets show that our approach is promising.

Original: A PPENDIX A  H YPER -P ARAMETERS AND N ETWORK A RCHITECTURES A. D ETAILS OF B ASELINES
Revised: A PPENDIX A N ETWORK A RCHITECTURES AND H YPER -P ARAMETERS A. M ULTIPLE N ETWORK A RCHITECTURE D ESIGNS FOR E NCODER AND D ECODER

Original: The details of all the baselines considered in our experiments are listed as follows: • ICA-LiNGAM assumes linear non-Gaussian additive model for data generating procedure andapplies independent component analysis to recover the weighted adjacency matrix. This method can usually achieve good performance on LiNGAM datasets.
Revised: There are a variety of neural network modules for encoder and decoder structures. Here we consider some representative modules: including , Multi-layer Perceptrons (MLP) module, an LSTM based recurrent neural network module, and the self-attention based encoder in the Transformer.

Original: However, it does not provide guarantee for linear Gaussian datasets.• NOTEARS recovers the causal graph by estimating the weighted adjacency matrix with the least squares loss and the smooth characterization for acyclicity
Revised: In addition, we use the original observation data as the state directly, i.e., no encoder module is used, to show the necessity of encoder, which is denoted as Null. MLP consists of 3 -layer feed-forward neural networks with 256 , 512 and 256 units.

Original: constraint.• DAG-GNN formulates causal discovery in the framework of variational autoencoder and optimizes a weighted adjacency matrix with the evidence lower bound and a modiﬁed smooth characterization on acyclicity as loss function.• GraN-DAG models the conditional distribution of each variable given its parents with feedforward neural networks.
Revised: The architecture of LSTM and Transformer are that introduced in Appendix A.2. The empirical results of CORL-2 on 30-node LG ER2 datasets are reported in Table 3.

Original: It uses the smooth directed acyclic constraint from NOTEARS to ﬁnd a DAG that maximizes the log-likelihood of the observed samples.• RL-BIC2 formulates the causal discovery as a one-step decision making process and uses score function and acyclic constraint from NOTEARS to calculate the reward for the recovered directed graph.• CAM conducts a greedy estimation procedure that starts with an empty DAG and adds at each iteration the edge ( v k , v j ) between nodes v k and v j thatcorresponds to the largest gain in log-likelihood. For a searched ordering, they prune it to the ﬁnal DAG by applying signiﬁcance testing of covariates.
Revised: We observe that LSTM decoder achieves a better performance than that of MLP decoder.

Original: They perform the preliminary neighborhood selection to reduce the ordering space size searched.• L1OBS performs heuristic search (greedy hill-climbing with tabu lists) through the space of topological orderings to search a ordering with the best score, then it uses L1 variable selection to pruning the searched ordering (fully-connected DAG) to the ﬁnal DAG.•
Revised: This shows that LSTM is more effective than MLP in sequential decision tasks. Besides, the overall performance of neural network encoder is better than that of Null, which shows that the data pre-processed by encoder module is necessary. Among all these encoders, Transformer encoder achieves the best results. Similar conclusion was drawned in Zhu et al.

Original: A* Lasso with a limited queue size incorporates a
Revised: We hypothesize that the performance of

Original: heuristic scheme into a DP-based method.
Revised: Transformer encoder is beneﬁt from the self-attention scheme.

Original: It ﬁrst prunes the search space by using A* Lasso, then it further pruning the search space by limiting the size of the priority queue in the OPEN list of A* Lasso.
Revised: [Table] LSTM Decoder TPR 0.94 (0.04) 0.88 (0.09) 0.97 (0.01) 0.99 (0.01)

Original: The queue size usually needs to be adjusted to balance the time cost and the quality of the solution.
Revised: SHD 20.6 (20.0) 29.0 (17.8) 8.6 (4.3) 4.4 (3.5)

Original: A. N ETWORK A RCHITECTURE AND H YPER -P ARAMETERS OF O UR M ETHOD
Revised: A. N ETWORK A RCHITECTURE AND H YPER -P ARAMETERS OF CORL

Original:  Both CORL-1 and CORL-2 use the actor-critic algorithm.
Revised: [Figure] Both CORL-1 and CORL-2 use the actor-critic algorithm.

Original: We use the Adam optimizer with learning rate 1 e − 4 and 1 e − 3 for actor and critic respectively.
Revised: We use the Adam optimizer with learning rate 1 e − 4 and 1 e − 3 for actor and critic respectively.

Original: The discount factor γ is set to 0 .
Revised: The discount factor γ is set to 0 .

Original: The actor  consists of an encoder and a decoder.
Revised: The actor [Figure] consists of an encoder and a decoder.

Original: We illustrate the neural network structure of the Transformer encoder used in our experiments in Figure 5.
Revised: We illustrate the neural network structure of the Transformer encoder used in our experiments in Figure 5.

Original: It consists of a feed-forward layer with 256 units and three blocks.
Revised: It consists of a feed-forward layer with 256 units and three blocks.

Original: Each block is composed of a multi-head attention network with 8 heads and 2 -layer feed-forward neural networks with 1024 and 256 units.
Revised: Each block is composed of a multi-head attention network with 8 heads and 2 -layer feed-forward neural networks with 1024 and 256 units.

Original: Each feed-forward layer is followed by a normalization layer.
Revised: Each feed-forward layer is followed by a normalization layer.

Original: Given a batch of observed samples with shape b × d × m , where b denotes the batch size, d the node number and m the number of observed data for each variable in a batch, the ﬁnal output of the encoder is a batch of embedded state with shape b ×
Revised: Given a batch of observed samples with shape b × d × n , where b denotes the batch size, d the node number and n the number of observed data for each variable in a batch, the ﬁnal output of the encoder is a batch of embedded state with shape b ×

Original: d
Revised: d

Original: × 256 .
Revised: × 256 .

Original: Structure_Learning/Causal_Discovery_RL

Original: [Figure]

Original: h

Original: [Figure]

Original: a d

Original: [Figure]

Original: … 𝐱 𝑗 … 𝐱 𝑑 𝑠… 𝑠

Original: 𝑗 … 𝑠 𝑑 [Figure] We illustrate the neural network structure of the decoder in Figure 6, which is mainly a LSTM similar to the decoder proposed by Vinyals et al. (2015).
Revised: We illustrate the neural network structure of the decoder in Figure 6, which is mainly a LSTM similar to the decoder proposed by Vinyals et al. (2015).

Original: The LSTM takes a state as input and outputs a embedding.
Revised: The LSTM takes a state as input and outputs a embedding.

Original: The embedding is mapped to the action space A by using some feed-forward neural networks, a soft-max module and the pointer mechanism (Vinyals et al., 2015).
Revised: The embedding is mapped to the action space A by using some feed-forward neural networks, a soft-max module and the pointer mechanism (Vinyals et al., 2015).

Original: The outputs of the encoder are processed as the initial hidden state h 0 of the decoder.
Revised: The outputs of the encoder are processed as the initial hidden state h 0 of the decoder.

Original: [-We pick the initial state as ˆ s 
-]
Revised: [-We pick the initial state as ˆ s 
-]

Original: = 1 d (cid:80) di

Original: =1 s i .

Original: The LSTM with 256 hidden units is used here.
Revised: The LSTM with 256 hidden units is used here.

Original: All of the feed-forward neural networks used in decoder have 256 units.
Revised: All of the feed-forward neural networks used in decoder have 256 units.

Original: The critic uses 3 -layer feed-forward neural networks with 512 , 256 and 1 units.
Revised: The critic uses 3 -layer feed-forward neural networks with 512 , 256 and 1 units.

Original: It takes a state ˆ s as input and outputs a predicted value for the current policy given state ˆ s .
Revised: It takes a state ˆ s as input and outputs a predicted value for the current policy given state ˆ s .

Original: For CORL-1, the critic needs to predict the score for each state.
Revised: For CORL-1, the critic needs to predict the score for each state.

Original: For CORL-2, the critic takes the initial state ˆ s 0 as input and outputs a predicted value directly for a complete ordering. 
Revised: For CORL-2, the critic takes the initial state ˆ s 0 as input and outputs a predicted value directly for a complete ordering. B B ASELINES AND D ATE S ETS B. D ETAILS OF B ASELINES

Revised: The details of all the baselines considered in our experiments are listed as follows:

Revised: • ICA-LiNGAM assumes linear non-Gaussian additive model for data generating procedure and applies independent component analysis to recover the weighted adjacency matrix.

Revised: This method can usually achieve good performance on LiNGAM datasets.

Original: B R ESULTS ON 30-N ODE , 50-N ODE AND 100-N ODE L I NGAM D ATASETS
Revised: However, it does not provide guarantee for linear Gaussian datasets.• NOTEARS recovers the causal graph by estimating the weighted adjacency matrix with the least squares loss and the smooth characterization for acyclicity constraint.• DAG-GNN formulates causal discovery in the framework of variational autoencoder and optimizes a weighted adjacency matrix with the evidence lower bound and a modiﬁed smooth characterization on acyclicity as loss function. • GraN-DAG models the conditional distribution of each variable given its parents with feedforward neural networks. It uses the smooth directed acyclic constraint from NOTEARS to ﬁnd a DAG that maximizes the log-likelihood of the observed samples.• RL-BIC2 formulates the causal discovery as a one-step decision making process and uses score function and acyclic constraint from NOTEARS to calculate the reward for the recovered directed graph.• CAM conducts a greedy estimation procedure that starts with an empty DAG and adds at each iteration the edge ( v k , v j ) between nodes v k and v j that corresponds to the largest gain in log-likelihood.

Revised: For a searched ordering, they prune it to the ﬁnal DAG by applying signiﬁcance testing of covariates.

Revised: They perform the preliminary neighborhood selection to reduce the ordering space size searched.• L1OBS performs heuristic search (greedy hill-climbing with tabu lists) through the space of topological orderings to search a ordering with the best score, then it uses L1 variable selection to pruning the searched ordering (fully-connected DAG) to the ﬁnal DAG.•

Revised: A* Lasso with a limited queue size incorporates a heuristic scheme into a dynamic programming based method.

Revised: It ﬁrst prunes the search space by using A* Lasso, then it further pruning the search space by limiting the size of the priority queue in the OPEN list of A* Lasso.

Revised: The queue size usually needs to be adjusted to balance the time cost and the quality of the solution.

Revised: B. D ETAILS ON D ATASETS G ENERATION

Revised: We generate synthetic datasets which vary along ﬁve dimensions: level of edge sparsity, graph type, number of nodes, causal functions and sample size.

Revised: We sampled 5 datasets of the required number examples for each task as follows: a ground truth DAG G is drawn randomly from either the

Revised: Erdös–Rényi (ER) or Scale-free (SF) graph model; then, the data is generated according to a speciﬁc sampling scheme.

Revised: Speciﬁcally, for Linear Gaussian (LG) case, we set h ∈ { 2 , 5 } and d ∈ { 30 , 50 , 100 } to obtain the

Revised: ER graph and SF graph (different types) with different levels of edge sparsity and different number of nodes, respectively.

Revised: Then 5 datasets of 3000 examples are generated for each task following

Revised: X = W T X

Revised: + (cid:15) , where W ∈ R d × d denotes the weight matrix which is obtained by assigning edge weights independently from Unif ([ − 2 , − 0 .

Revised: 5] ∪ [0 . 5 , 2]) .

Revised: Note that (cid:15) ’s are standard Gaussian noises with equal variances for each variable so as to LG data model is identiﬁable (Peters & Bühlmann, 2014).

Revised: For LiNGAM data model, the datasets are generated in the similar way with LG but the sampling for (cid:15) are different.

Revised: The non-Gaussian noises are obtained by following Shimizu et al.

Revised: (2006) which passes the noise samples from Gaussian distribution through a power nonlinearity to make them non-Gaussian.

Revised: LiNGAM is identiﬁable shown in Shimizu et al.

Revised: Another data generating process is GP.

Revised: We ﬁrst obtain graph with different density and different number of nodes.

Revised: Then the datasets with different sample sizes are generated following X j = f j ( Pa ( X j ))

Revised: + (cid:15) j with jointly independent Gaussian noises for all j where the function f

Revised: j is a function sampled from a GP with radial basis function kernel of bandwidth one and normal Gaussian noises.

Revised: This setting is known to be identiﬁable according to Peters et al.

Revised: Note that due to the efﬁciency of the reward calculation, we only experimented on nonlinear data models of up to 30 nodes.

Revised: C T OTAL N UMBER OF E PISODES B EFORE C ONVERGENCE

Revised: Table 4 reports the total number of episodes required for CORL-2 and RL-BIC2 to be converged, averaged over ﬁve datasets.

Revised: Note that the episodic reward is evaluated once per episode.

Revised: CORL formulates causal discovery as a multi-stage decision process and we observe that CORL performs fewer episodes than RL-BIC2 before the episode reward converges.

Revised: We suspect that due to a signiﬁcant reduction in the size of action space, the model complexity of the RL policy is reduced thus leading to higher sample efﬁciency.

Revised: Some runtimes about them are also provided here (CORLtotal runtime ≈ 15 minutes against RL-BIC2 ≈ 3 hours for 30-node ER2 graphs, ≈ 4 hours against ≈ 14 hour for 50-node ER2 graphs, and CORL-2 ≈ 7 hours for 100-node ER2 graphs).

Revised: We set the maximal runtime up to 24 hours, but RL-BIC2 did not converge within that time on 100-node graphs, hence we did not report it here.

Revised: Note that these runtime may be signiﬁcantly reduced by parallelizing the evaluation of reward.

Revised: [Table]

Revised: D A DDITIONAL R ESULTS ON L INEAR G AUSSIAN D ATASETS

Revised: The results for 50 -node LG data models are presented in Table 5.

Revised: The conclusion is similar to the 30and 100 -node experiments.

Revised: The results of ICA-LiNGAM, GraN-DAG and CAM on LG data models are presented in Table 6.

Revised: Their performances do not compare favorably to CORL-1 nor CORL-2 in

Revised: LG datasets.

Revised: It is not surprising that ICA-LiNGAM does not perform well because the algorithm is speciﬁcally designed for non-Gaussian noise and does not provide guarantee for LG data models.

Revised: We hypothesize that CAM’s poor performance on LG data models is because it uses nonlinear regression instead of linear regression.

Revised: As for GraN-DAG, it uses 2 -layer feed-forward neural networks to model the causal relationships, which may not be able to learn a good linear relationship in this experiment.

Original:  Here, we report the empirical results on 30-node, 50-node and 100-node LiNGAM datasets in Table 2.
Revised: [Table] E R ESULTS ON 30-, 50- AND 100-N ODE L I NGAM D ATASETS Here, we report the empirical results on 30-, 50- and 100-node LiNGAM datasets in Table 7.

Original: The observed samples are generated according to the same procedure with linear Gaussian datasets.
Revised: The observed samples are generated according to the same procedure with linear Gaussian datasets (see

Revised: Appendix B.2 for details).

Original: The non-Gaussian noise is obtained by passing the noise samples from Gaussian distribution through a power nonlinearity to make them non-Gaussian.
Revised: The non-Gaussian noise is obtained by passing the noise samples from Gaussian distribution through a power nonlinearity to make them non-Gaussian.

Original: For L1OBS, we increased the authors’ recommended number of evaluations 2500 to 10000 .
Revised: For L1OBS, we increased the authors’ recommended number of evaluations 2500 to 10000 .

Original: For A* Lasso, we set the queue size to 10, 500 and 1000, and report the best result for all of these different parameter settings.
Revised: For A* Lasso, we set the queue size to 10, 500 and 1000, and report the best result for all of these different parameter settings.

Original: The results of L1OBS and A*[Table] [Table] Lasso reported here are that of pruning using our pruning method.
Revised: The results of L1OBS and A* Lasso reported here are that of pruning using our pruning method.

Original: For other baselines, we pick the recommended hyper-parameters.
Revised: For other baselines, we pick the recommended hyper-parameters.

Original: Among all these algorithms [Table] ICA-LiNGAM can recover the true graph on most of the LiNGAM data models.
Revised: Among all these algorithms, ICA-LiNGAM can recover the true graph on most of the LiNGAM data models.

Original: This is because ICA-LiNGAM is speciﬁcally designed for non-Gaussian noise.
Revised: This is because ICA-LiNGAM is speciﬁcally designed for non-Gaussian noise.

Original: CORLand CORL-2 achieve consistently good results than other baselines. 
Revised: CORLand CORL-2 achieve consistently good results than other baselines. [Table] [Table]

Original: C R ESULTS ON 20-N ODE GP D ATASETS WITH D IFFERENT S AMPLE S IZES
Revised: F R ESULTS ON 20-N ODE GP D ATASETS WITH D IFFERENT S AMPLE S IZES

Original: We take the 20-node GP data models as an example to show the performance of our method on different sample numbers.
Revised: We take the 20-node GP data models as an example to show the performance of our method on different sample numbers.

Original: We set n = 4 to get ER4 graphs and generate data by using them.
Revised: We set h = 4 to get ER4 graphs and generate data by using them.

Original: We illustrate the empirical results in Table 3.
Revised: We illustrate the empirical results in Table 8.

Original: Since previous experiments have shown that CORLis slightly better than CORL-1, we only report the results of CORL-2 here.
Revised: Since previous experiments have shown that CORLis slightly better than CORL-1, we only report the results of CORL-2 here.

Original: CAM as the most competitive baseline, we also report its results on these datasets.
Revised: CAM as the most competitive baseline, we also report its results on these datasets.

Original: TPR reported here is calculated based on the variable ordering.
Revised: TPR reported here is calculated based on the variable ordering.

Original: We can see that, as the sample size decreases, CORL-2 ends up outperforming CAM.
Revised: We can see that, as the sample size decreases, CORL-2 ends up outperforming CAM.

Original: We believe this is because CORL-2 beneﬁts from the exploratory ability of RL.
Revised: We believe this is because CORL-2 beneﬁts from the exploratory ability of RL.

Original: D CORL-1 WITH A P RETRAINED M ODEL
Revised: G CORL-1 WITH A P RETRAINED M ODEL

Original: We show the training reward curves of CORL-1 and CORL-1-pretrain which is CORL-1 based on a pretrained model in Figure 7.
Revised: We show the training reward curves of CORL-1 and CORL-1-pretrain which is CORL-1 based on a pretrained model in Figure 7.

Revised: To obtain a pretraining model with good generalization ability, we combine various types data described in Appendix B.2 with different levels of edge sparsity, graph

Revised: [Table]

Original: We train a policy model by supervised learning on thedatasets mixed with non-linear data models and LiNGAM data models, then we useit as the pretrainedpolicy model for linear Gaussian data models.
Revised: [Figure] type and number of nodes and data generating process to construct observation samples. Next, we train a policy model by supervised learning on the mixed datasets. Finally, we use the pretrained model as a start point on the task that has never been seen before during pretraining.

Original: Although the pretrained model is trained on other types of datasets, it can still accelerate the training as the initial model on the linear Gaussian dataset task.
Revised: From Figure 7, we can observe that although the pretrained model is trained on other types of datasets, it can still accelerate the training as the initial model on the LG dataset task.

Original: This shows that our policy model may learn some implicit knowledge across different tasks.
Revised: This shows that our policy model may learn some implicit knowledge across different tasks.

Original: [Figure]