Original: A MORTIZED C AUSAL D ISCOVERY : L EARNING TO I NFER C AUSAL G RAPHS FROM T IME -S ERIES D ATA
Revised: A MORTIZED C AUSAL D ISCOVERY : L EARNING TO I NFER C AUSAL G RAPHS FROM T IME -S ERIES D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Standard causal discovery methods must ﬁt a new model whenever they encounter samples from a new underlying causal graph.
Revised: Standard causal discovery methods must ﬁt a new model whenever they encounter samples from a new underlying causal graph.

Original: However, these samples often share relevant information – for instance, the dynamics describing the effects of causal relations – which is lost when following this approach.
Revised: However, these samples often share relevant information – for instance, the dynamics describing the effects of causal relations – which is lost when following this approach.

Original: We propose Amortized
Revised: We propose Amortized

Original: Causal Discovery, a novel framework that leverages such shared dynamics to learn to infer causal relations from time-series data.
Revised: Causal Discovery, a novel framework that leverages such shared dynamics to learn to infer causal relations from time-series data.

Original: This enables us to train a single, amortized model that infers causal relations across samples with different underlying causal graphs, and thus makes use of the information that is shared.
Revised: This enables us to train a single, amortized model that infers causal relations across samples with different underlying causal graphs, and thus makes use of the information that is shared.

Original: We demonstrate experimentally that this approach, implemented as a variational model, leads to signiﬁcant improvements in causal discovery performance, and show how it can be extended to perform well under hidden confounding.
Revised: We demonstrate experimentally that this approach, implemented as a variational model, leads to signiﬁcant improvements in causal discovery performance, and show how it can be extended to perform well under hidden confounding.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Inferring causal relations in observational time-series is central to many ﬁelds of scientiﬁc inquiry (Berzuini et al., 2012; Spirtes et al., 2000).
Revised: Inferring causal relations in observational time-series is central to many ﬁelds of scientiﬁc inquiry (Berzuini et al., 2012; Spirtes et al., 2000).

Original: Suppose you want to analyze fMRI data, which measures the activity of different brain regions over time — how can you infer the (causal) inﬂuence of one brain region on another?
Revised: Suppose you want to analyze fMRI data, which measures the activity of different brain regions over time — how can you infer the (causal) inﬂuence of one brain region on another?

Original: This question is addressed by the ﬁeld of causal discovery (Glymour et al., 2019).
Revised: This question is addressed by the ﬁeld of causal discovery (Glymour et al., 2019).

Original: Methods within this ﬁeld allow us to infer causal relations from observational data - when interventions (e.g. randomized trials) are infeasible, unethical or too expensive.
Revised: Methods within this ﬁeld allow us to infer causal relations from observational data - when interventions (e.g. randomized trials) are infeasible, unethical or too expensive.

Original: In time-series, the assumption that causes temporally precede their effects enables us to discover causal relations in observational data (Peters et al., 2017); with approaches relying on conditional independence tests (Entner and Hoyer, 2010), scoring functions (Chickering, 2002), or deep learning (Tank et al., 2018).
Revised: In time-series, the assumption that causes temporally precede their effects enables us to discover causal relations in observational data (Peters et al., 2017); with approaches relying on conditional independence tests (Entner and Hoyer, 2010), scoring functions (Chickering, 2002), or deep learning (Tank et al., 2018).

Original: All of these methods assume that samples share a single underlying causal graph and reﬁt a new model whenever this assumption does not hold.
Revised: All of these methods assume that samples share a single underlying causal graph and reﬁt a new model whenever this assumption does not hold.

Original: However, samples with different underlying causal graphs may share relevant information such as the dynamics describing the effects of causal relations.
Revised: However, samples with different underlying causal graphs may share relevant information such as the dynamics describing the effects of causal relations.

Original: fMRI test subjects may have varying brain connectivity but the same underlying neurochemistry; social networks may have differing structure but comparable interpersonal relationships; different stocks may relate differently to one another but obey similar market forces.
Revised: fMRI test subjects may have varying brain connectivity but the same underlying neurochemistry; social networks may have differing structure but comparable interpersonal relationships; different stocks may relate differently to one another but obey similar market forces.

Original: Despite a range of relevant applications, inferring causal relations across samples with different underlying causal graphs is as of yet largely unexplored.
Revised: Despite a range of relevant applications, inferring causal relations across samples with different underlying causal graphs is as of yet largely unexplored.

Original: In this paper, we propose a novel causal discovery framework for time-series that embraces this aspect: Amortized Causal Discovery (Fig. 1).
Revised: In this paper, we propose a novel causal discovery framework for time-series that embraces this aspect: Amortized Causal Discovery (Fig. 1).

Original: In this framework, we learn to infer causal relations across samples with different underlying causal graphs but shared dynamics.
Revised: In this framework, we learn to infer causal relations across samples with different underlying causal graphs but shared dynamics.

Original: We achieve this by separating the causal relation prediction from the modeling of their dynamics: an amortized encoder predicts the edges in the causal graph, and a decoder models the dynamics of the system under the predicted causal relations.
Revised: We achieve this by separating the causal relation prediction from the modeling of their dynamics: an amortized encoder predicts the edges in the causal graph, and a decoder models the dynamics of the system under the predicted causal relations.

Original: This setup allows us to pool statistical strength across samples and to achieve signiﬁcant improvements in performance with additional training data.
Revised: This setup allows us to pool statistical strength across samples and to achieve signiﬁcant improvements in performance with additional training data.

Original: It also allows us to infer causal relations in previously unseen samples without reﬁtting our model.
Revised: It also allows us to infer causal relations in previously unseen samples without reﬁtting our model.

Original: Additionally, we show that Amortized Causal Discovery allows us to improve robustness under hidden confounding by modeling the unobserved variables with the amortized encoder.
Revised: Additionally, we show that Amortized Causal Discovery allows us to improve robustness under hidden confounding by modeling the unobserved variables with the amortized encoder.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We formalize Amortized Causal Discovery (ACD), a novel framework for causal discovery in time-series, in which we learn to infer causal relations from samples with different underlying causal graphs but shared dynamics.
Revised: • We formalize Amortized Causal Discovery (ACD), a novel framework for causal discovery in time-series, in which we learn to infer causal relations from samples with different underlying causal graphs but shared dynamics.

Original: • We propose a variational model for ACD, applicable to multi-variate, non-linear data.
Revised: • We propose a variational model for ACD, applicable to multi-variate, non-linear data.

Original: • We present experiments demonstrating the effectiveness of this model on a range of causal discovery datasets, both in the fully observed setting and under hidden confounding.
Revised: • We present experiments demonstrating the effectiveness of this model on a range of causal discovery datasets, both in the fully observed setting and under hidden confounding.

Original:  B ACKGROUND : G RANGER C AUSALITY
Revised:  B ACKGROUND : G RANGER C AUSALITY

Original: Granger causality (Granger, 1969) is one of the most commonly used approaches to infer causal relations from observational time-series data.
Revised: Granger causality (Granger, 1969) is one of the most commonly used approaches to infer causal relations from observational time-series data.

Original: Its central assumption is that causes precede their effects: if the prediction of the future of time-series Y can be improved by knowing past elements of time-series X , then X “Granger causes” Y .
Revised: Its central assumption is that causes precede their effects: if the prediction of the future of time-series Y can be improved by knowing past elements of time-series X , then X “Granger causes” Y .

Original: Originally, Granger causality was deﬁned for linear relations; we follow the more recent deﬁnition of Tank et al.
Revised: Originally, Granger causality was deﬁned for linear relations; we follow the more recent deﬁnition of Tank et al.

Original: (2018) for non-linear Granger causality:
Revised: (2018) for non-linear Granger causality:

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: Non-Linear Granger Causality :
Revised: Non-Linear Granger Causality :

Original: Given N stationary time-series x = { x 1 , ... x N } across time-steps t = { 1 , ..., T } and a non-linear autoregressive function g j , such that
Revised: Given N stationary time-series x = { x 1 , ... x N } across time-steps t = { 1 , ..., T } and a non-linear autoregressive function g j , such that

Original: [Equation]
Revised: [Equation]

Original: (1) where x ≤ t j = ( ..., x t − 1 j , x tj ) denotes the present and past of series j and ε tj represents independent noise.
Revised: (1) where x ≤ t j = ( ..., x t − 1 j , x tj ) denotes the present and past of series j and ε tj represents independent noise.

Original: In this setup, time-series i Granger causes j , if g j is not invariant to x ≤ t i , i.e. if ∃ x (cid:48)≤ t i (cid:54) = x ≤ t i : g j ( x ≤ t 1 , ..., x (cid:48)≤ t i , ..., x ≤
Revised: In this setup, time-series i Granger causes j , if g j is not invariant to x ≤ t i , i.e. if ∃ x (cid:48)≤ t i (cid:54) = x ≤ t i : g j ( x ≤ t 1 , ..., x (cid:48)≤ t i , ..., x ≤

Original: [Equation]
Revised: [Equation]

Original: Granger causal relations are equivalent to causal relations in the underlying directed acyclic graph if all relevant variables are observed and no instantaneous 1 connections exist (Peters et al., 2013; 2017,
Revised: Granger causal relations are equivalent to causal relations in the underlying directed acyclic graph if all relevant variables are observed and no instantaneous 1 connections exist (Peters et al., 2013; 2017,

Original: Theorem 10.1).
Revised: Theorem 10.1).

Original: Many methods for Granger causal discovery, including vector autoregressive (Hyvärinen et al., 2010) and more recent deep learning-based approaches (Khanna and Tan, 2020; Tank et al., 2018; Wu et al., 2020), can be encapsulated by a particular framework:
Revised: Many methods for Granger causal discovery, including vector autoregressive (Hyvärinen et al., 2010) and more recent deep learning-based approaches (Khanna and Tan, 2020; Tank et al., 2018; Wu et al., 2020), can be encapsulated by a particular framework:

Original: 1. Deﬁne a function f θ (an MLP in Tank et al.
Revised: 1. Deﬁne a function f θ (an MLP in Tank et al.

Original: a linear model in Hyvärinen et al.
Revised: a linear model in Hyvärinen et al.

Original: (2010)), which learns to predict the next time-step of the test sequence x . 2.
Revised: (2010)), which learns to predict the next time-step of the test sequence x . 2.

Original: Fit
Revised: Fit

Original: f θ to x by minimizing some loss L : θ (cid:63) =
Revised: f θ to x by minimizing some loss L : θ (cid:63) =

Original: argmin θ L ( x , f θ ) .
Revised: argmin θ L ( x , f θ ) .

Original: Apply some ﬁxed function h (e
Revised: Apply some ﬁxed function h (e

Original: g. thresholding) to the learned parameters to produce the Granger causal graph estimate for x : ˆ G x = h ( θ (cid:63) ) .
Revised: g. thresholding) to the learned parameters to produce the Granger causal graph estimate for x : ˆ G x = h ( θ (cid:63) ) .

Original: For instance, Tank et al.
Revised: For instance, Tank et al.

Original: (2018) infer the Granger causal relations through examination of the weights θ (cid:63) : if all outgoing weights w ij between time-series i and j are zero, then i does not Granger-cause j .
Revised: (2018) infer the Granger causal relations through examination of the weights θ (cid:63) : if all outgoing weights w ij between time-series i and j are zero, then i does not Granger-cause j .

Original: The shortcoming of this approach is that, when we have S samples x 1 , . . .
Revised: The shortcoming of this approach is that, when we have S samples x 1 , . . .

Original: , x S with different underlying causal graphs, the parameters θ must be optimized separately for each of them.
Revised: , x S with different underlying causal graphs, the parameters θ must be optimized separately for each of them.

Original: As a result, methods within this framework cannot take advantage of the information that might be shared between samples.
Revised: As a result, methods within this framework cannot take advantage of the information that might be shared between samples.

Original: This motivates us to question: can we amortize this process?
Revised: This motivates us to question: can we amortize this process?

Original:  A MORTIZED C AUSAL D ISCOVERY
Revised:  A MORTIZED C AUSAL D ISCOVERY

Original: We propose Amortized Causal Discovery (ACD), a framework in which we learn to infer causal relations across samples with different underlying causal graphs but shared dynamics.
Revised: We propose Amortized Causal Discovery (ACD), a framework in which we learn to infer causal relations across samples with different underlying causal graphs but shared dynamics.

Original: To illustrate:
Revised: To illustrate:

Original: Suppose you want to infer synaptic connections (i.e. causal relations) between neurons based on their spiking behaviour.
Revised: Suppose you want to infer synaptic connections (i.e. causal relations) between neurons based on their spiking behaviour.

Original: You are given a set of S recordings (i.e. samples), each containing N time-series representing the ﬁring of N individual neurons.
Revised: You are given a set of S recordings (i.e. samples), each containing N time-series representing the ﬁring of N individual neurons.

Original: Even though you might record across different populations of neurons with different wiring, the dynamics of how neurons connected by synapses inﬂuence one another stays the same.
Revised: Even though you might record across different populations of neurons with different wiring, the dynamics of how neurons connected by synapses inﬂuence one another stays the same.

Original: ACD takes advantage of such shared dynamics to improve the prediction of causal relations.
Revised: ACD takes advantage of such shared dynamics to improve the prediction of causal relations.

Original: It can be summarized as follows:
Revised: It can be summarized as follows:

Original: 1. Deﬁne an encoding function f φ which learns to infer Granger causal relations of any sample x i in the training set X train .
Revised: 1. Deﬁne an encoding function f φ which learns to infer Granger causal relations of any sample x i in the training set X train .

Original: Deﬁne a decoding function f θ which learns to predict the next time-step of the samples under the inferred causal relations.
Revised: Deﬁne a decoding function f θ which learns to predict the next time-step of the samples under the inferred causal relations.

Original: Fit f φ and f θ to X train by minimizing some loss L : f φ (cid:63) , f θ (cid:63) =
Revised: Fit f φ and f θ to X train by minimizing some loss L : f φ (cid:63) , f θ (cid:63) =

Original: argmin f φ ,f θ L ( X train , f φ , f θ ) .
Revised: argmin f φ ,f θ L ( X train , f φ , f θ ) .

Original: For any given test sequence x test , simply output the Granger causal graph estimate ˆ G x test : ˆ G x test =
Revised: For any given test sequence x test , simply output the Granger causal graph estimate ˆ G x test : ˆ G x test =

Original: f φ (cid:63) ( x test ) .
Revised: f φ (cid:63) ( x test ) .

Original: By dividing the model into two parts, an encoder and a decoder, ACD can use the activations of f φ (cid:63) to infer causal structure.
Revised: By dividing the model into two parts, an encoder and a decoder, ACD can use the activations of f φ (cid:63) to infer causal structure.

Original: This increases the ﬂexibility of our approach greatly compared to methods that use the learned weights θ (cid:63) such as the prior Granger causal discovery methods described in
Revised: This increases the ﬂexibility of our approach greatly compared to methods that use the learned weights θ (cid:63) such as the prior Granger causal discovery methods described in

Original: Section 2.
Revised: Section 2.

Original: In this section, we describe our framework in more detail, and provide a probabilistic implementation thereof.
Revised: In this section, we describe our framework in more detail, and provide a probabilistic implementation thereof.

Original: We also extend our approach to model hidden confounders.
Revised: We also extend our approach to model hidden confounders.

Original: Preliminaries We begin with a dataset X
Revised: Preliminaries We begin with a dataset X

Original: = { x s } Ss =1 of S samples, where each sample x s consists of N stationary time-series x s = { x s, 1 , . . .
Revised: = { x s } Ss =1 of S samples, where each sample x s consists of N stationary time-series x s = { x s, 1 , . . .

Original: , x s,N } across timesteps t = { 1 , ..., T } .
Revised: , x s,N } across timesteps t = { 1 , ..., T } .

Original: We denote the t -th time-step of the i -th time-series of x s as x ts,i .We assume there is a directed acyclic graph
Revised: We denote the t -th time-step of the i -th time-series of x s as x ts,i .We assume there is a directed acyclic graph

Original: G 1: T s =
Revised: G 1: T s =

Original: {V 1: T s , E 1: T s } underlying the generative process of each sample.
Revised: {V 1: T s , E 1: T s } underlying the generative process of each sample.

Original: This is a structural causal model (SCM) (Pearl, 2009).
Revised: This is a structural causal model (SCM) (Pearl, 2009).

Original: Its endogenous (observed) variables are vertices v ts,i ∈ V 1: T s for each time-series i and each time-step t .
Revised: Its endogenous (observed) variables are vertices v ts,i ∈ V 1: T s for each time-series i and each time-step t .

Original: Every set of incoming edges to an endogenous variable deﬁnes inputs to a deterministic function g ts,i which determines that variable’s value 2 .
Revised: Every set of incoming edges to an endogenous variable deﬁnes inputs to a deterministic function g ts,i which determines that variable’s value 2 .

Original: The edges are deﬁned by ordered pairs of vertices E 1: T s =
Revised: The edges are deﬁned by ordered pairs of vertices E 1: T s =

Original: { ( v ts,i , v t (cid:48) s,j ) } , which we make two assumptions about:
Revised: { ( v ts,i , v t (cid:48) s,j ) } , which we make two assumptions about:

Original: No edges are instantaneous ( t = t (cid:48) ) or go back in time.
Revised: No edges are instantaneous ( t = t (cid:48) ) or go back in time.

Original: Thus, t < t (cid:48) for all edges.
Revised: Thus, t < t (cid:48) for all edges.

Original: Edges are invariant to time.
Revised: Edges are invariant to time.

Original: Thus, if ( v ts,i , v t + k s,j ) ∈ E 1: T s , then ∀ 1 ≤ t (cid:48) ≤ T − k : ( v t (cid:48) s,i , v t (cid:48)
Revised: Thus, if ( v ts,i , v t + k s,j ) ∈ E 1: T s , then ∀ 1 ≤ t (cid:48) ≤ T − k : ( v t (cid:48) s,i , v t (cid:48)

Original: + k s,j ) ∈ E 1: T s .
Revised: + k s,j ) ∈ E 1: T s .

Original: The associated structural equations g ts,i are invariant to time as well, i.e. g ts,i = g t
Revised: The associated structural equations g ts,i are invariant to time as well, i.e. g ts,i = g t

Original: (cid:48)
Revised: (cid:48)

Original: s,i ∀ t, t (cid:48) .
Revised: s,i ∀ t, t (cid:48) .

Original: The ﬁrst assumption states that causes temporally precede their effects and makes causal relations identiﬁable from observational data, when no hidden confounders are present (Peters et al., 2013; 2017, Theorem 10.1).
Revised: The ﬁrst assumption states that causes temporally precede their effects and makes causal relations identiﬁable from observational data, when no hidden confounders are present (Peters et al., 2013; 2017, Theorem 10.1).

Original: The second simpliﬁes modeling: it is a fairly general assumption which allows us to deﬁne dynamics that govern all time-steps (Eq.
Revised: The second simpliﬁes modeling: it is a fairly general assumption which allows us to deﬁne dynamics that govern all time-steps (Eq.

Original: Throughout this paper, we are interested in discovering the summary graph G s =
Revised: Throughout this paper, we are interested in discovering the summary graph G s =

Original: {V s , E s } (Peters et al., 2017).
Revised: {V s , E s } (Peters et al., 2017).

Original: It consists of vertices v s,i ∈
Revised: It consists of vertices v s,i ∈

Original: V s for each time-series i in sample s , and has directed edges whenever they exist in E 1: T s at any time-step,
Revised: V s for each time-series i in sample s , and has directed edges whenever they exist in E 1: T s at any time-step,

Original: i.e. E s =
Revised: i.e. E s =

Original: ( v s,i , v s,j ) | ∃ t, t (cid:48) : ( v ts,i , v t (cid:48) s,
Revised: ( v s,i , v s,j ) | ∃ t, t (cid:48) : ( v ts,i , v t (cid:48) s,

Original: j ) ∈
Revised: j ) ∈

Original: E 1: T s } .
Revised: E 1: T s } .

Original: Note that while G 1: T s is acyclic (due to the ﬁrst assumption above), the summary graph G s may contain (self-)cycles.
Revised: Note that while G 1: T s is acyclic (due to the ﬁrst assumption above), the summary graph G s may contain (self-)cycles.

Original: Amortized Causal Discovery
Revised: Amortized Causal Discovery

Original: The key assumption for Amortized Causal Discovery is that there exists some ﬁxed function g that describes the dynamics of all samples x s ∈ X given their past
Revised: The key assumption for Amortized Causal Discovery is that there exists some ﬁxed function g that describes the dynamics of all samples x s ∈ X given their past

Original: and
Revised: graph G s :

Original: [Equation]
Revised: [Equation]

Original: There are two variables in this data-generating process that we would like to model: the causal graph
Revised: There are two variables in this data-generating process that we would like to model: the causal graph

Original: G s that is speciﬁc to sample x s , and the dynamics g that are shared across all samples.
Revised: G s that is speciﬁc to sample x s , and the dynamics g that are shared across all samples.

Original: This separation between the causal graph and the dynamics allows us to divide our model accordingly: we introduce an amortized causal discovery encoder f φ which learns to infer a causal graph G s given the sample x s , and a dynamics decoder f θ that learns to approximate g :
Revised: This separation between the causal graph and the dynamics allows us to divide our model accordingly: we introduce an amortized causal discovery encoder f φ which learns to infer a causal graph G s given the sample x s , and a dynamics decoder f θ that learns to approximate g :

Original: [Equation]
Revised: [Equation]

Original: We formalize Amortized Causal Discovery (ACD) as follows.
Revised: We formalize Amortized Causal Discovery (ACD) as follows.

Original: Let G be the domain of all possible summary graphs on x s : G s ∈ G .
Revised: Let G be the domain of all possible summary graphs on x s : G s ∈ G .

Original: Let X be the domain of any single step, partial or full, observed sequence: x ts , x ≤ t s , x s ∈ X .
Revised: Let X be the domain of any single step, partial or full, observed sequence: x ts , x ≤ t s , x s ∈ X .

Original: The model consists of two components: a causal discovery encoder f φ : X → G which infers a causal graph for each input sample, and a decoder f θ : X × G → X which models the dynamics.
Revised: The model consists of two components: a causal discovery encoder f φ : X → G which infers a causal graph for each input sample, and a decoder f θ : X × G → X which models the dynamics.

Original: This model is optimized with a sample-wise loss (cid:96) :
Revised: This model is optimized with a sample-wise loss (cid:96) :

Original: X × X → R which scores how well the decoder models the true dynamics of x s , and a regularization term r : G → R on the inferred graphs.
Revised: X × X → R which scores how well the decoder models the true dynamics of x s , and a regularization term r : G → R on the inferred graphs.

Original: For example, this function r may enforce sparsity by penalizing graphs with more edges.
Revised: For example, this function r may enforce sparsity by penalizing graphs with more edges.

Original: Note, that our formulation of the graph prediction problem is unsupervised: we do not have access to the true underlying graph G s .
Revised: Note, that our formulation of the graph prediction problem is unsupervised: we do not have access to the true underlying graph G s .

Original: Then, given some dataset X train with S samples, we optimize:
Revised: Then, given some dataset X train with S samples, we optimize:

Original: [Equation]
Revised: [Equation]

Original: where L ( X train , φ, θ ) =
Revised: where L ( X train , φ, θ ) =

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Revised: See Appendix B for a proof of the consistency of the loss (cid:96) and a discussion on regularization r .

Original: Once we have completed optimization, we can perform causal graph prediction on any new input test sample x test in two ways – we can feed x test into the amortized encoder and take its output as the predicted edges (Eq. 6); or we can instantiate our estimate ˆ G test ∈ G which will be our edge predictions, and ﬁnd the edges which best explain the observed sequence x test by minimizing the (learned) decoding loss with respect to ˆ G test , which we term Test-Time Adaptation (TTA) (Eq. 7):
Revised: Once we have completed optimization, we can perform causal graph prediction on any new input test sample x test in two ways – we can feed x test into the amortized encoder and take its output as the predicted edges (Eq. 6); or we can instantiate our estimate ˆ G test ∈ G which will be our edge predictions, and ﬁnd the edges which best explain the observed sequence x test by minimizing the (learned) decoding loss with respect to ˆ G test , which we term Test-Time Adaptation (TTA) (Eq. 7):

Original: [Equation]
Revised: [Equation]

Original: By separating the prediction of causal relations from the modeling of their dynamics, ACD yields a number of beneﬁts.
Revised: By separating the prediction of causal relations from the modeling of their dynamics, ACD yields a number of beneﬁts.

Original: ACD can learn to infer causal relations across samples with different underlying causal graphs, and it can infer causal relations in previously unseen test samples without reﬁtting (Eq.
Revised: ACD can learn to infer causal relations across samples with different underlying causal graphs, and it can infer causal relations in previously unseen test samples without reﬁtting (Eq.

Original: By generalizing across samples, it can improve causal discovery performance with increasing training data size.
Revised: By generalizing across samples, it can improve causal discovery performance with increasing training data size.

Original: We can replace either f φ or f θ with ground truth annotations, or simulate the outcome of counterfactual causal relations.
Revised: We can replace either f φ or f θ with ground truth annotations, or simulate the outcome of counterfactual causal relations.

Original: Additionally, ACD can be applied in the standard causal discovery setting, where only a single causal graph underlies all samples, by replacing the amortized encoder f φ with an estimated graph ˆ G (or distribution over G ) in Eq.
Revised: Additionally, ACD can be applied in the standard causal discovery setting, where only a single causal graph underlies all samples, by replacing the amortized encoder f φ with an estimated graph ˆ G (or distribution over G ) in Eq.

Original: 3. A P ROBABILISTIC I MPLEMENTATION OF ACD
Revised: 3. A P ROBABILISTIC I MPLEMENTATION OF ACD

Original: We take a probabilistic approach to ACD and model the functions f φ and f θ using variational inference (Fig. 2).
Revised: We take a probabilistic approach to ACD and model the functions f φ and f θ using variational inference (Fig. 2).

Original: We amortize the encoder f φ with a function q φ ( z | x ) , which outputs a distribution over z representing the predicted edges ˆ E in the causal graph; and we learn a decoder p θ ( x | z ) which probabilistically models the dynamics of the time-series under the predicted causal relations.
Revised: We amortize the encoder f φ with a function q φ ( z | x ) , which outputs a distribution over z representing the predicted edges ˆ E in the causal graph; and we learn a decoder p θ ( x | z ) which probabilistically models the dynamics of the time-series under the predicted causal relations.

Original: We choose a negative log-likelihood for the decoder loss (cid:96) and a KL-Divergence to a prior distribution over G for the regularizer r .
Revised: We choose a negative log-likelihood for the decoder loss (cid:96) and a KL-Divergence to a prior distribution over G for the regularizer r .

Original: As a result, our loss function L is a variational lower bound:
Revised: As a result, our loss function L is a variational lower bound:

Original: [Equation]
Revised: [Equation]

Original: Encoder The encoder q φ ( z | x ) applies a graph neural network f enc ,φ (Gilmer et al., 2017; Kipf and
Revised: Encoder The encoder q φ ( z | x ) applies a graph neural network f enc ,φ (Gilmer et al., 2017; Kipf and

Original: Welling, 2017;
Revised: Welling, 2017;

Original: Li et al., 2016; Scarselli et al., 2008) to the input, which propagates information across afully connected graph G = {V, E} .
Revised: Li et al., 2016; Scarselli et al., 2008) to the input, which propagates information across a connected graph G = {V

Original: This graph includes vertices v i ∈ V for each time-series i , and each pair of vertices ( v i , v j ) is connected by an edge.
Revised: E

Original: [Equation]
Revised: [Equation]

Original: To enable us to backpropagate through the samples of the discrete distribution q φ ( z ij | x ) , during training, we relax it by adding Gumbel distributed noise g (Jang et al., 2017; Maddison et al., 2017):
Revised: To enable us to backpropagate through the samples of the discrete distribution q φ ( z ij | x ) , during training, we relax it by adding Gumbel distributed noise g (Jang et al., 2017; Maddison et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: The output z ij of the encoder represents the predicted edges ˆ E Enc in the causal graph ˆ G Enc .
Revised: The output z ij of the encoder represents the predicted edges ˆ E Enc in the causal graph ˆ G Enc .

Original: We consider the possibility that there are n E different edge-types expressing causal relationships; for instance, inhibitory or excitatory synaptic connections.
Revised: We consider the possibility that there are n E different edge-types expressing causal relationships; for instance, inhibitory or excitatory synaptic connections.

Original: Then, more speciﬁcally, z ij,e = 1 expresses that there is a directed edge of type e from time-series i to j , where e ∈ { 0 , . . .
Revised: Then, more speciﬁcally, z ij,e = 1 expresses that there is a directed edge of type e from time-series i to j , where e ∈ { 0 , . . .

Original: , n E − 1 } .
Revised: , n E − 1 } .

Original: Decoder The decoder p θ ( x | z ) models the dynamics of the time-series under the predicted causal relations.
Revised: Decoder The decoder p θ ( x | z ) models the dynamics of the time-series under the predicted causal relations.

Original: It uses both the predicted causal relations z ij and the feature vectors of the time-series at the current time-step t , x t = { x t 1 , ... x tN } as its input.
Revised: It uses both the predicted causal relations z ij and the feature vectors of the time-series at the current time-step t , x t = { x t 1 , ... x tN } as its input.

Original: First, it propagates information along the predicted edges by applying a neural network f e , using the zero function for f 0 :
Revised: First, it propagates information along the predicted edges by applying a neural network f e , using the zero function for f 0 :

Original: [Equation]
Revised: [Equation]

Original: Then, the decoder accumulates the incoming messages to each node and applies a neural network f v to predict the change between the current and the next time-step:
Revised: Then, the decoder accumulates the incoming messages to each node and applies a neural network f v to predict the change between the current and the next time-step:

Original: [Equation]
Revised: [Equation]

Original: In other words, the decoder predicts ∆ˆ x t , which is added to the current value of the time-series to yield the prediction for the next time-step ˆ x t +1 = x t + ∆ˆ x t .
Revised: In other words, the decoder predicts ∆ˆ x t , which is added to the current value of the time-series to yield the prediction for the next time-step ˆ x t +1 = x t + ∆ˆ x t .

Original: Prediction of Causal Relations In order to align our model with the philosophy of Granger
Revised: Prediction of Causal Relations In order to align our model with the philosophy of Granger

Original: Causality, we include a “no edge”-type edge function: If the encoder predicts the “no edge”-type edge e = 0 by setting z ij, 0 = 1 , the decoder uses the zero function and no information is propagated from time-series i to j (Eq.
Revised: Causality, we include a “no edge”-type edge function: If the encoder predicts the “no edge”-type edge e = 0 by setting z ij, 0 = 1 , the decoder uses the zero function and no information is propagated from time-series i to j (Eq.

Original: Due to this, time-series i will Granger cause the decoder-predicted time-series j only when the edge is predicted to exist (see Appendix A).
Revised: Due to this, time-series i will Granger cause the decoder-predicted time-series j only when the edge is predicted to exist (see Appendix A).

Original: Hence, by the same logic that justiﬁes prior Granger causal work (Section 2), we expect the predicted edges to correspond to Granger causal relations.
Revised: Hence, by the same logic that justiﬁes prior Granger causal work (Section 2), we expect the predicted edges to correspond to Granger causal relations.

Original: Finally, since we assume no hidden confounders and no instantaneous edges, these Granger causal relations will correspond to relations in the underlying SCM (Peters et al., 2017, Theorem 10.3).
Revised: Finally, since we assume no hidden confounders and no instantaneous edges, these Granger causal relations will correspond to relations in the underlying SCM (Peters et al., 2017, Theorem 10.3).

Original: 3. H IDDEN C ONFOUNDING
Revised: 3. H IDDEN C ONFOUNDING

Original: Hidden confounders are a critical problem in the time-series context: when they exist, Granger causality is not guaranteed to correspond to the true causal graph anymore (Peters et al., 2017,
Revised: Hidden confounders are a critical problem in the time-series context: when they exist, Granger causality is not guaranteed to correspond to the true causal graph anymore (Peters et al., 2017,

Original: Theorem 10.3) 3 .
Revised: Theorem 10.3) 3 .

Original: Inspired by proxy-based methods from causal inference (e.g. Louizos et al.
Revised: Inspired by proxy-based methods from causal inference (e.g. Louizos et al.

Original: (2017), see Section 4), we present a method for applying ACD to the hidden confounding setting.
Revised: (2017), see Section 4), we present a method for applying ACD to the hidden confounding setting.

Original: First, we extend the amortized encoder q φ ( z | x ) to predict an additional variable.
Revised: First, we extend the amortized encoder q φ ( z | x ) to predict an additional variable.

Original: Then, we encourage this variable to model the hidden confounder by applying a structural bias – depending on the type of unobserved variable that we want to model, its predicted value is utilized differently by the remaining model.
Revised: Then, we encourage this variable to model the hidden confounder by applying a structural bias – depending on the type of unobserved variable that we want to model, its predicted value is utilized differently by the remaining model.

Original: The decoder remains responsible for modeling the dynamics, and now also processes the predictions for the unobserved variable.
Revised: The decoder remains responsible for modeling the dynamics, and now also processes the predictions for the unobserved variable.

Original: While this setup might not allow us to identify the hidden confounders, the data-driven approach underlying ACD can beneﬁt our model: by pooling the statistical strength across samples with different underlying causal graphs, our model can learn to mitigate the effects of the hidden confounders.
Revised: While this setup might not allow us to identify the hidden confounders, the data-driven approach underlying ACD can beneﬁt our model: by pooling the statistical strength across samples with different underlying causal graphs, our model can learn to mitigate the effects of the hidden confounders.

Original: We consider two types of hidden confounders: 1) a temperature variable that confounds all endogenous variables by inﬂuencing the strength of their causal relations.
Revised: We consider two types of hidden confounders: 1) a temperature variable that confounds all endogenous variables by inﬂuencing the strength of their causal relations.

Original: This temperature is sampled separately for each sample, and remains constant throughout each sample.
Revised: This temperature is sampled separately for each sample, and remains constant throughout each sample.

Original: 2) a hidden variable that behaves just like the observed variables, i.e. it may affect or be affected by the observed variables through the same causal relations, and its value changes across time.
Revised: 2) a hidden variable that behaves just like the observed variables, i.e. it may affect or be affected by the observed variables through the same causal relations, and its value changes across time.

Original: In both cases, we extend the encoder to predict this hidden variable, and feed that prediction into the decoder.
Revised: In both cases, we extend the encoder to predict this hidden variable, and feed that prediction into the decoder.

Original: We provide more details in
Revised: We provide more details in

Original: Section 5.2.
Revised: Section 5.2.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A range of approaches to causal discovery in both temporal and non-temporal data exist (Heinze-Deml et al., 2018; Peters et al., 2017; Spirtes et al., 2000).
Revised: A range of approaches to causal discovery in both temporal and non-temporal data exist (Heinze-Deml et al., 2018; Peters et al., 2017; Spirtes et al., 2000).

Original: One common class is constraint-based , relying on conditional independence testing to uncover an underlying DAG structure or equivalence class (Spirtes et al., 2000).
Revised: One common class is constraint-based , relying on conditional independence testing to uncover an underlying DAG structure or equivalence class (Spirtes et al., 2000).

Original: These methods predict a single graph ˆ G (or equivalence class) for all samples.
Revised: These methods predict a single graph ˆ G (or equivalence class) for all samples.

Original: There is no notion of ﬁtting a dynamics model for time-series methods in this class (Entner and
Revised: There is no notion of ﬁtting a dynamics model for time-series methods in this class (Entner and

Original: Hoyer, 2010).
Revised: Hoyer, 2010).

Original: Another common class of methods for causal discovery is score-based (Bengio et al., 2019; Chickering, 2002).
Revised: Another common class of methods for causal discovery is score-based (Bengio et al., 2019; Chickering, 2002).

Original: Here, a score function h is chosen, and the methods perform a search through graph space to optimize this score, i.e. ˆ G = argmin G h ( G ) .
Revised: Here, a score function h is chosen, and the methods perform a search through graph space to optimize this score, i.e. ˆ G = argmin G h ( G ) .

Original: Our proposed decoder-based inference (Eq.
Revised: Our proposed decoder-based inference (Eq.

Original: (7)) can be seen as score-based causal discovery with a learned score function L ◦ f θ (cid:63) .
Revised: (7)) can be seen as score-based causal discovery with a learned score function L ◦ f θ (cid:63) .

Original: A third class of methods ﬁts a (possibly regularized) dynamics model f and then analyzes its form to produce a causal graph estimate, by using linear dynamics (Hyvärinen et al., 2010), recurrent models (Khanna and Tan, 2020; Nauta et al., 2019; Tank et al., 2018), or other deep-learning based approaches (Lachapelle et al., 2019; Wu et al., 2020; Zheng et al., 2020).
Revised: A third class of methods ﬁts a (possibly regularized) dynamics model f and then analyzes its form to produce a causal graph estimate, by using linear dynamics (Hyvärinen et al., 2010), recurrent models (Khanna and Tan, 2020; Nauta et al., 2019; Tank et al., 2018), or other deep-learning based approaches (Lachapelle et al., 2019; Wu et al., 2020; Zheng et al., 2020).

Original: See Section 2 for discussion.
Revised: See Section 2 for discussion.

Original: Other approaches to causal discovery in temporal data use independence or additivity assumptions (Eichler, 2012; Peters et al., 2013).
Revised: Other approaches to causal discovery in temporal data use independence or additivity assumptions (Eichler, 2012; Peters et al., 2013).

Original: A number of works have explored the idea of jointly learned causal structure across examples, including a range of papers in the setting where a number of related datasets are collected, possibly with different columns (Dhir and Lee, 2020; Huang et al., 2019; 2020;
Revised: A number of works have explored the idea of jointly learned causal structure across examples, including a range of papers in the setting where a number of related datasets are collected, possibly with different columns (Dhir and Lee, 2020; Huang et al., 2019; 2020;

Original: Shimizu, 2012; Tillman and Eberhardt, 2014).
Revised: Shimizu, 2012; Tillman and Eberhardt, 2014).

Original: Li et al.
Revised: Li et al.

Original: (2018) proposes learning a linear mixed effects model across samples, and concurrent work explores amortized deep learning of differing types of causal structure (Ke et al., 2020; Li et al., 2020).
Revised: (2018) proposes learning a linear mixed effects model across samples, and concurrent work explores amortized deep learning of differing types of causal structure (Ke et al., 2020; Li et al., 2020).

Original: Very few papers systematically study the hidden confounding setting.
Revised: Very few papers systematically study the hidden confounding setting.

Original: Some empirical work shows that encoder-based models with enough proxies (variables caused by hidden confounders) can improve causal inference under hidden confounding (Louizos et al., 2017; Parbhoo et al., 2020), and theoretical work proves the identiﬁability of latent variables from proxies under some assumptions (Allman et al., 2009; Kruskal, 1977).
Revised: Some empirical work shows that encoder-based models with enough proxies (variables caused by hidden confounders) can improve causal inference under hidden confounding (Louizos et al., 2017; Parbhoo et al., 2020), and theoretical work proves the identiﬁability of latent variables from proxies under some assumptions (Allman et al., 2009; Kruskal, 1977).

Original: Several works have used graph neural networks (Battaglia et al., 2016;
Revised: Several works have used graph neural networks (Battaglia et al., 2016;

Original: Kipf et al., 2018; Santoro et al., 2017) or attention mechanisms (Fuchs et al., 2019;
Revised: Kipf et al., 2018; Santoro et al., 2017) or attention mechanisms (Fuchs et al., 2019;

Original: Goyal et al., 2019; Van Steenkiste et al., 2018;
Revised: Goyal et al., 2019; Van Steenkiste et al., 2018;

Original: Vaswani et al., 2017) to infer relations between time-series.
Revised: Vaswani et al., 2017) to infer relations between time-series.

Original: Alet et al.
Revised: Alet et al.

Original: (2019) propose a meta-learning algorithm to additionally model unobserved variables.
Revised: (2019) propose a meta-learning algorithm to additionally model unobserved variables.

Original: While these approaches model object relations in a number of ways, they are not explicitly designed to infer causal graphical structure.
Revised: While these approaches model object relations in a number of ways, they are not explicitly designed to infer causal graphical structure.

Original: [Figure]
Revised: [Figure]

Original: The probabilistic implementation of ACD is based on the Neural Relational Inference (NRI) model (Kipf et al., 2018), extended by a special zero-function to allow for a causal interpretation of the inferred edges, new inference methods using test-time adaptation and a new algorithm to handle confounders.
Revised: The probabilistic implementation of ACD is based on the Neural Relational Inference (NRI) model (Kipf et al., 2018), extended by a special zero-function to allow for a causal interpretation of the inferred edges, new inference methods using test-time adaptation and a new algorithm to handle confounders.

Original: Moreover, the new model is applied to a different problem than NRI, namely (Granger) causal discovery, and we show that it outperforms the current state of the art for this type of problem.
Revised: Moreover, the new model is applied to a different problem than NRI, namely (Granger) causal discovery, and we show that it outperforms the current state of the art for this type of problem.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Implementation We measure causal discovery performance by area under the receiver operator curve (AUROC) of predicted edge probabilities over test samples.
Revised: Implementation We measure causal discovery performance by area under the receiver operator curve (AUROC) of predicted edge probabilities over test samples.

Original: We compare to recurrent models (Khanna and Tan (2020); Tank et al.
Revised: We compare to recurrent models (Khanna and Tan (2020); Tank et al.

Original: (2018)), and a mutual-information (MI) based model by Wu et al.
Revised: (2018)), and a mutual-information (MI) based model by Wu et al.

Original: and several baselines implemented by those authors, including MI (unmodiﬁed), transfer entropy (Schreiber, 2000), and a linear Granger causality.
Revised: and several baselines implemented by those authors, including MI (unmodiﬁed), transfer entropy (Schreiber, 2000), and a linear Granger causality.

Original: More details in Appendix B.
Revised: More details in Appendix C.

Original: 5. F ULLY O BSERVED A MORTIZED C AUSAL D ISCOVERY
Revised: 5. F ULLY O BSERVED A MORTIZED C AUSAL D ISCOVERY

Original: We test ACD on three datasets: two fully-observed physics simulations (Kuramoto and Particles) and the Netsim dataset of simulated fMRI data (Smith et al., 2011).
Revised: We test ACD on three datasets: two fully-observed physics simulations (Kuramoto and Particles) and the Netsim dataset of simulated fMRI data (Smith et al., 2011).

Original: Note, in contrast to the physics simulations used in Kipf et al.
Revised: Note, in contrast to the physics simulations used in Kipf et al.

Original: (2018), we generate data with asymmetric connectivity matrices to represent causal relations.
Revised: (2018), we generate data with asymmetric connectivity matrices to represent causal relations.

Original: [Table]
Revised: [Table]

Original: First, we test our method on the Kuramoto dataset , which contains ﬁve 1-D time-series of phase-coupled oscillators (Kuramoto, 1975).
Revised: First, we test our method on the Kuramoto dataset , which contains ﬁve 1-D time-series of phase-coupled oscillators (Kuramoto, 1975).

Original: We ﬁnd that ACD greatly outperforms all approaches for Granger causal discovery that we compare against (Table 1).
Revised: We ﬁnd that ACD greatly outperforms all approaches for Granger causal discovery that we compare against (Table 1).

Original: In contrast to these approaches, ACD achieves this result without ﬁtting to the test samples.
Revised: In contrast to these approaches, ACD achieves this result without ﬁtting to the test samples.

Original: Additionally, we ﬁnd that ACD can indeed utilize samples with different underlying causal graphs – its performance improves steadily with increasing training data size (Fig. 3).
Revised: Additionally, we ﬁnd that ACD can indeed utilize samples with different underlying causal graphs – its performance improves steadily with increasing training data size (Fig. 3).

Original: Nonetheless, it is also applicable to the low-data regime: when applying ACD with test-time adaptation (TTA), it requires less than 50 training samples to outperform all previous approaches.
Revised: Nonetheless, it is also applicable to the low-data regime: when applying ACD with test-time adaptation (TTA), it requires less than 50 training samples to outperform all previous approaches.

Original: We note that the baseline performance here is worse than presented elsewhere in the literature — this is because we evaluate accuracy without considering prediction of self-connectivity, as self-connections are the easiest type to predict.
Revised: We note that the baseline performance here is worse than presented elsewhere in the literature — this is because we evaluate accuracy without considering prediction of self-connectivity, as self-connections are the easiest type to predict.

Original: In our second experiment, we apply ACD to the particles dataset .
Revised: In our second experiment, we apply ACD to the particles dataset .

Original: This dataset models ﬁve particles that move around a two-dimensional space, with some particles inﬂuencing others uni-directionally by pulling them with a spring.
Revised: This dataset models ﬁve particles that move around a two-dimensional space, with some particles inﬂuencing others uni-directionally by pulling them with a spring.

Original: Since all previous methods were intended for one-dimensional time-series,
Revised: Since all previous methods were intended for one-dimensional time-series,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: we were unable to evaluate them in this domain.
Revised: we were unable to evaluate them in this domain.

Original: ACD, on the other hand, is readily applicable to higher-dimensional data, and performs almost perfectly on this dataset with 0.999 AUROC .
Revised: ACD, on the other hand, is readily applicable to higher-dimensional data, and performs almost perfectly on this dataset with 0.999 AUROC .

Original: In both experiments, causal relation prediction with the learned encoder (Enc - Eq.
Revised: In both experiments, causal relation prediction with the learned encoder (Enc - Eq.

Original: (6)) performs best in the high-data regime, while test-time adaptation (TTA - Eq.
Revised: (6)) performs best in the high-data regime, while test-time adaptation (TTA - Eq.

Original: (7)) improves the performance in low-data settings (Fig. 3).
Revised: (7)) improves the performance in low-data settings (Fig. 3).

Original: This beneﬁt of TTA can be largely attributed to two effects.
Revised: This beneﬁt of TTA can be largely attributed to two effects.

Original: First,
Revised: First,

Original: TTA closes the amortization gap of the encoder (Cremer et al., 2018).
Revised: TTA closes the amortization gap of the encoder (Cremer et al., 2018).

Original: Second, TTA overcomes the encoder’s overﬁtting on the training data (as seen in the training curves in Appendix B.2) by being adapted to the individual test samples.
Revised: Second, TTA overcomes the encoder’s overﬁtting on the training data (as seen in the training curves in Appendix C.2) by being adapted to the individual test samples.

Original: On the particles dataset, initializing TTA with the encoder’s prediction (Enc+TTA) improves over a random initialization (TTA) as the encoder improves; but we do not observe this effect on the Kuramoto dataset.
Revised: On the particles dataset, initializing TTA with the encoder’s prediction (Enc+TTA) improves over a random initialization (TTA) as the encoder improves; but we do not observe this effect on the Kuramoto dataset.

Original: [Table]
Revised: [Table]

Original: Finally, we apply ACD to the Netsim dataset (Smith et al., 2011) of simulated fMRI data.
Revised: Finally, we apply ACD to the Netsim dataset (Smith et al., 2011) of simulated fMRI data.

Original: Here, the task is to infer the underlying connectivity between 15 brain regions acrosssamples.
Revised: Here, the task is to infer the underlying connectivity between 15 brain regions acrosssamples.

Original: A single graph underlies all samples, allowing us to demonstrate ACD’s applicability to the classical setting.
Revised: A single graph underlies all samples, allowing us to demonstrate ACD’s applicability to the classical setting.

Original: We replace the amortized encoder q φ ( z | x ) with a global latent distribution q ( z ) , optimize it through the decoder, and then use test-time adaptation (TTA).
Revised: We replace the amortized encoder q φ ( z | x ) with a global latent distribution q ( z ) , optimize it through the decoder, and then use test-time adaptation (TTA).

Original: Even though our model cannot beneﬁt from its datadriven design here, it performs comparably to methods that are intended for use in the singlegraph setting (Table 2).
Revised: Even though our model cannot beneﬁt from its datadriven design here, it performs comparably to methods that are intended for use in the singlegraph setting (Table 2).

Original: 5. A MORTIZED C AUSAL D ISCOVERY UNDER H IDDEN C ONFOUNDING 5.2. L ATENT T EMPERATURE
Revised: 5. A MORTIZED C AUSAL D ISCOVERY UNDER H IDDEN C ONFOUNDING 5.2. L ATENT T EMPERATURE

Original: In this experiment, we use the particles dataset and vary an unobserved temperature variable, which modulates how strongly the particles exert force on each other – higher temperatures result in stronger forces and a more chaotic system.
Revised: In this experiment, we use the particles dataset and vary an unobserved temperature variable, which modulates how strongly the particles exert force on each other – higher temperatures result in stronger forces and a more chaotic system.

Original: For each x s , we sample an independent temperature c ∼ Categorical ([ α 2 , α, 2 α ]) from a categorical distribution with α ∈ R and equal probabilities.
Revised: For each x s , we sample an independent temperature c ∼ Categorical ([ α 2 , α, 2 α ]) from a categorical distribution with α ∈ R and equal probabilities.

Original: We predict this unobserved temperature by extending the amortized encoder with an additional latent variable which models a uniform distribution.
Revised: We predict this unobserved temperature by extending the amortized encoder with an additional latent variable which models a uniform distribution.

Original: Then, we add a KL-Divergence between this posterior and a uniform prior on the interval [0 , 4 α ] to our variational loss.
Revised: Then, we add a KL-Divergence between this posterior and a uniform prior on the interval [0 , 4 α ] to our variational loss.

Original: To allow for learning in this setting, we introduce an inductive bias: we usea decoder which matches the true dynamics g given the predicted temperature and causal relations. See Appendix C.1 for more details and additional results.
Revised: To allow for learning in this setting, we introduce an inductive bias: we use

Original: Results Fig.
Revised: Results Fig.

Original: 4 shows the causal discovery results across different values of α .
Revised: 4 shows the causal discovery results across different values of α .

Original: ACD enhanced with an additional latent variable ( Latent ) outperforms both tested baselines across all temperatures:
Revised: ACD enhanced with an additional latent variable ( Latent ) outperforms both tested baselines across all temperatures:

Original: Mean , which uses the same ground-truth decoder as Latent and ﬁxes the decoder temperature to be the mean of the categorical distribution, and None , which does not model c explicitly and trains
Revised: Mean , which uses the same ground-truth decoder as Latent and ﬁxes the decoder temperature to be the mean of the categorical distribution, and None , which does not model c explicitly and trains

Original: an
Revised: an

Original: MLP decoder.
Revised: MLP decoder.

Original: Additionally, this method achieves high predictive performance on the unobserved temperature variable: for α = 2, temperature prediction obtains 0.888 R 2 , 0.966 AUROC and 0.
Revised: Additionally, this method achieves high predictive performance on the unobserved temperature variable: for α = 2, temperature prediction obtains 0.888 R 2 , 0.966 AUROC and 0.

Original: accuracy.
Revised: accuracy.

Original: These results indicate that we can model an unobserved temperature variable, and thus improve robustness under hidden confounding.
Revised: These results indicate that we can model an unobserved temperature variable, and thus improve robustness under hidden confounding.

Original: 5.2. U NOBSERVED T IME -S ERIES
Revised: 5.2. U NOBSERVED T IME -S ERIES

Original: , we treat one of the original time series in the particles dataset as unobserved.
Revised: , we treat one of the original time series in the particles dataset as unobserved.

Original: It exhibits the same dynamics as the observed time-series, evolving and causally inﬂuencing others the same way as before.
Revised: It exhibits the same dynamics as the observed time-series, evolving and causally inﬂuencing others the same way as before.

Original: This challenging setting has received little attention in the literature;
Revised: This challenging setting has received little attention in the literature;

Original: Alet et al.
Revised: Alet et al.

Original: (2019) tackled it with mixed success.
Revised: (2019) tackled it with mixed success.

Original: We model the unobserved time-series by extending the amortized encoder with an additional latent variable and applying a suitable structural bias: the latent prediction z tu for time-steps t = { 1 , ..., T } is treated in the same way as the observed time-series x .
Revised: We model the unobserved time-series by extending the amortized encoder with an additional latent variable and applying a suitable structural bias: the latent prediction z tu for time-steps t = { 1 , ..., T } is treated in the same way as the observed time-series x .

Original: Its entire trajectory is used by the encoder to predict causal relations, and its value at the current time-step is fed into the decoder.
Revised: Its entire trajectory is used by the encoder to predict causal relations, and its value at the current time-step is fed into the decoder.

Original: See Appendix C.2 for more details and additional results.
Revised: See Appendix D.2 for more details and additional results.

Original: Results Fig.
Revised: Results Fig.

Original: 5 shows how the causal discovery AUROC depends on the number of observed time-series directly inﬂuenced by the unobserved one.
Revised: 5 shows how the causal discovery AUROC depends on the number of observed time-series directly inﬂuenced by the unobserved one.

Original: When this number is zero, all tested approaches perform the same.
Revised: When this number is zero, all tested approaches perform the same.

Original: With growing numbers of inﬂuenced time-series, the baselines that either ignore the missing time-series ( None ) or impute its value with the average of the observed time-series over time ( Mean ) deteriorate strongly.
Revised: With growing numbers of inﬂuenced time-series, the baselines that either ignore the missing time-series ( None ) or impute its value with the average of the observed time-series over time ( Mean ) deteriorate strongly.

Original: In contrast, the proposed ACD with a Latent variable stays closer to the performance of the fully Observed baseline.
Revised: In contrast, the proposed ACD with a Latent variable stays closer to the performance of the fully Observed baseline.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, it also improves the future trajectory prediction of the observed time-series.
Revised: 6, it also improves the future trajectory prediction of the observed time-series.

Original: A Supervised baseline that uses the (usually unavailable) ground-truth trajectories to optimize the prediction of the unobserved time-series, improves only slightly over our approach.
Revised: A Supervised baseline that uses the (usually unavailable) ground-truth trajectories to optimize the prediction of the unobserved time-series, improves only slightly over our approach.

Original: These results indicate that ACD can use latent variables to improve robustness to unobserved time-series.
Revised: These results indicate that ACD can use latent variables to improve robustness to unobserved time-series.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce ACD, a framework for causal discovery in time-series data which can leverage the information that is shared across samples.
Revised: In this paper, we introduce ACD, a framework for causal discovery in time-series data which can leverage the information that is shared across samples.

Original: We provide a probabilistic implementation and demonstrate signiﬁcant performance gains when predicting causal relations, even under hidden confounding.
Revised: We provide a probabilistic implementation and demonstrate signiﬁcant performance gains when predicting causal relations, even under hidden confounding.

Original: Exciting future directions include interventions, more ﬂexible graph structures, or methods that adapt dynamically to the type of hidden confounder at hand.
Revised: Exciting future directions include interventions, more ﬂexible graph structures, or methods that adapt dynamically to the type of hidden confounder at hand.