L ATENT C ONVERGENT C ROSS M APPING
Anonymous authors
Paper under double-blind review
A BSTRACT
Discovering causal structures of temporal processes is a major tool of scientiﬁc inquiry because it helps us better understand and explain the mechanisms driving a phenomenon of interest, thereby facilitating analysis, reasoning, and synthesis for such systems.
However, accurately inferring causal structures within a phenomenon based on observational data only is still an open problem.
Indeed, this type of data usually consists in short time series with missing or noisy values for which causal inference is increasingly difﬁcult.
In this work, we propose a method to uncover causal relations in chaotic dynamical systems from short, noisy and sporadic time series (that is, incomplete observations at infrequent and irregular intervals) where the classical convergent cross mapping (CCM) fails.
Our method works by learning a Neural ODE latent process modeling the state-space dynamics of the time series and by checking the existence of a continuous map between the resulting processes.
We provide theoretical analysis and show empirically that Latent-CCM can reliably uncover the true causal pattern, unlike traditional methods.
I NTRODUCTION
Inferring a right causal model of a physical phenomenon is at the heart of scientiﬁc inquiry.
It is fundamental to how we understand the world around us and to predict the impact of future interventions (Pearl, 2009).
Correctly inferring causal pathways helps us reason about a physical system, anticipate its behavior in previously unseen conditions, design changes to achieve some objective, or synthesize new systems with desirable behaviors.
As an example, in medicine, causality inference could allow predicting whether a drug will be effective for a speciﬁc patient, or in climatology, to assess human activity as a causal factor in climate change.
Causal mechanisms are best uncovered by making use of interventions because this framework leads to an intuitive and robust notion of causality.
However, there is a signiﬁcant need to identify causal dependencies when only observational data is available, because such data is more readily available as it is more practical and less costly to collect ( e.g. , relying on observational studies when interventional clinical trials are not yet available).
However, real-world data arising from less controlled environment than, for instance, clinical trials poses many challenges for analysis.
Confounding and selection bias come into play, which bias standard statistical estimators.
If no intervention is possible, some causal conﬁgurations cannot be identiﬁed.
Importantly, with real-world data comes the major issue of missing values.
In particular, when collecting longitudinal data, the resulting time series are often sporadic : sampling is irregular in time and across dimensions leading to varying time intervals between observations of a given variable and typically multiple missing observations at any given time.
This problem is ubiquitous in various ﬁelds, such as healthcare (De Brouwer et al., 2019), climate science (Thomson, 1990), or astronomy (Cuevas-Tello et al., 2010).
A key problem in causal inference is to assess whether one temporal variable is causing another or is merely correlated with it.
From assessing causal pathways for neural activity (Roebroeck et al., 2005) to ecology (Sugihara et al., 2012) or healthcare, it is a necessary step to unravel underlying generating mechanisms.
A common way to infer causal direction between two temporal variables is to use Granger causality (Granger, 1969), which deﬁnes “predictive causality” in terms of the predictability of one time series from the other.
A key requirement of Granger causality is then separability (i.e., that information about causes are not contained in the caused variable itself).
This assumption holds in purely stochastic linear systems, but fails in more general cases (such as weakly coupled nonlinear dynamical systems)
(Sugihara et al., 2012).
To address this nonseparability issue,
Sugihara et al.
(Sugihara et al., 2012) introduced the Convergent Cross Mapping (CCM) method, which is based on the theory of chaotic dynamical systems, particularly on Takens’ theorem.
This method has been applied successfully in various ﬁelds such as ecology, climatology (Wang et al., 2018), and neuroscience (Schiecke et al., 2015).
However, as the method relies on embedding the time series under study with time lags, it is highly sensitive to missing values and usually requires long uninterrupted time series.
This method is thus not applicable in settings with repeated short sporadic time series, despite their occurrence in many practical situations.
To address this important limitation, we propose to learn the causal dependencies between time series by checking the existence of convergent cross mappings between latent processes of those time series.
Using a joint model across all segments of sporadically observed time series and forcing the model to learn the inherent dynamic of the data, we show that our method can detect causal relationship from short and sporadic time series, without computing delay embeddings.
To learn a continuous time latent representation of the system’s state-space, we leverage GRU-ODE-Bayes (De Brouwer et al., 2019), a recently introduced ﬁltering method that extends the Neural ODE model (Chen et al., 2018).
Importantly for causal inference, the ﬁltering nature of the model makes sure no future information can leak into the past.
We then check the existence of continuous maps between the learnt latent representations and infer the causal direction accordingly.
In a series of increasingly challenging test cases, our method accurately detects the correct causal dependencies with high conﬁdence, even when fed very few observations, and outperforms competing methods such as multi-spatial CCM or CCM with multivariate Gaussian process interpolation.
[Figure]
dotted line) from the latent process of Y ( H Y ) to the latent process of X ( H X ).
R ELATED WORK
CCM to address failure of Granger causality.
Granger causality (Granger, 1969) provided the ﬁrst signiﬁcant framework to infer causal dependencies from time series.
Relying on predictability between dynamical systems, it was extended to account for different limitations, such as nonlinearity (Chen et al., 2004) or instantaneous relationships (Schiatti et al., 2015).
However, the assumption of separability of information between causative and caused variables leads to the failure of the Granger paradigm for a signiﬁcant number of time series coupling scenarios (Sugihara et al., 2012) (see
Appendix D for a revealing worked out example).
Convergent Cross Mapping, a technique based on nonlinear state space reconstruction was introduced to tackle this issue (Sugihara et al., 2012).
Recently, several works have proposed extensions of CCM, such as the extended CCM, to address issues such as synchrony (Ye et al., 2015) or to improve the discrimination of the confounding case (Benk˝o et al., 2018).
Synchrony occurs when one time series can be expressed as a function of the other ( e.g. Y ( t )
=
φ ( X ( t ) ) and attractors of both dynamical systems become homeomorphic to each other (Rulkov et al., 1995).
This occurs when coupling between two chaotic system is too strong.
Confounding, on the other hand, occurs when two variables are causally driven by a third one.
In general we say that X confounds the relation between Y and Z if X causes both Y and Z .
Huang et al.
(2020) also proposed to predict directly the driving time series from the driven one with reservoir computing, bypassing the delay embedding step, making it more robust to noise.
However, those methods still require long regularly sampled time series.
Causality for short or sporadic time series.
Short time series are very common in practice and there has been some work proposing to learn causality from short time series relying on state space reconstruction.
Ma et al.
(2014) proposed a method for short, fully observed, unique time series.
Multi-spatial CCM (Clark et al., 2015), considered the problem of inferring causality from several short fully observed snippets of the same dynamical system by computing delay embeddings compatible with the lengths of the time series and aggregating them.
In comparison, on top of addressing irregular sampling, our approach computes more informative state-space representations by sharing a model across all segments.
Techniques to infer causal direction from incomplete time series have also been proposed, but all are relying on the Granger causality framework, which limits their applicability to separable dynamical systems.
They use direct partial correlations on regularly sampled data (but with missing values)
(Elsegai, 2019) or generalizations of similarity measures for sporadic time series (Bahadori & Liu, 2012).
To the best of our knowledge, this is the ﬁrst work investigating the identiﬁcation of causal dependencies from short sporadic time series using state-space reconstruction.
M ETHOD
We consider the problem of inferring a causal dependency between two temporal variables from several segments of their multivariate time series X [ t ] ∈ R d X and Y [ t ] ∈ R d Y .
We assume that
X [ t ] and Y [ t ] have been generated by an unknown dynamical system.
In this work, we refer to the dynamical system of a time varying variable X as the smallest dynamical system that fully describes the dynamics of X .
As an example, let’s consider the following system of ODEs representing the dynamics of X and Y :
[Equation]
The dynamical system of X is given by Equation (1).
On the other hand, the dynamical system of
Y is Equation (1) + (2) as Equation (1) is required to describe the dynamics of Y .
To account for the more general and most frequent case, we consider those time series are only observed in segments of ﬁnite duration.
X [ t ] and Y [ t ] then consist of collections of N short time series ( X 1 [ t ] ,..., X N [ t ]) ) and ( Y 1 [ t ] ,..., Y N [ t ]) ) respectively.
Importantly, each segment of X and
Y is observed concomitantly.
To proceed with a lighter notation, we’ll drop the superscript when referring to a segment of time series.
Each of those time series is also sporadic namely the are not regularly sampled and not all dimensions are observed each time.
In this work, we deﬁne the notion of causality by considering the equations of the dynamical system as a structural causal model.
In this framework, X causes Y if p ( Y | do ( X ))
(cid:54) =
P ( Y ) where do ( X ) is an intervention on X (Pearl, 2009).
Then, if X causes Y , X is part of the dynamical system of Y ( X is required to describe the dynamics of Y ).
In the case of the example described by Equationsand 2, X causes Y if g ( · ) is not a constant function.
3. C ONVERGENT C ROSS M APPING AND T AKENS ’ THEOREM
CCM aims at discovering the causal direction between temporal variables in dynamical systems by checking if the state-space dynamics of their time series can be recovered from one another.
As shown above, if X causes Y , X is then contained in the dynamical system of Y and it should be possible to recover a representation of the dynamical system of X from the dynamical system of Y .
A common way to obtain a representation of a dynamical system from its time series relies on
Takens’ embedding theorem (Takens, 1981).
Let X [ t ] ∈ R d X be issued from a chaotic dynamical system that has a strange attractor M with box-counting dimension d M , where we deﬁne an attractor as the manifold toward which the state of a chaotic dynamical system tends to evolve.
The dynamics of this system are speciﬁed by a ﬂow on
M , φ ( · ) ( · ) : R × M → M , where φ τ ( M t ) =
M t + τ and M t stands for the point on the manifold at time index t .
This ﬂow is encoded in the ODE of the system.
The observed time series X [ t ] is then obtained through an observation function
f obs ( · ) : X
[ t ] = f obs ( M t ) .
Takens’ theorem then states that a delay embedding Φ with delay τ and embedding dimension k
[Equation]
is an embedding of the strange attractor M if k > 2 d M and α : R d M
→ R is a twice-differentiable observation function.
More speciﬁcally, the embedding map Φ is a diffeomorphism between the original strange attractor manifold M and a shadow attractor manifold M (cid:48) generated by the delay embeddings.
Under these assumptions, one can then theoretically reconstruct the original time series from the delay embedding.
The simplest observation function α consists in simply taking one of the dimensions of observations of the dynamical system.
In this case, writing X i [ t ] as the i -th dimension of X [ t ] , Takens’ theorem ensures that there is a diffeomorphism between the original attractor manifold of the full dynamical system and the shadow manifold M (cid:48) that would be generated by X (cid:48)
[ t ] =
( X i [ t ] , X i [ t − τ ] , . . .
, X i [ t − kτ ]) .
To see how this theorem can be used to infer the causal direction, let us consider the manifold M Z of the joint dynamical system resulting of the concatenation of
X [ t ] and Y [ t ] .
We then generate two shadow manifolds M (cid:48)
X and M (cid:48)
Y from the delay embeddings
X (cid:48)
[ t ] =
( X i [ t ] , X i [ t − τ ] , . . .
, X i [ t − kτ ]) and Y (cid:48)
[ t ] : ( Y j [ t ] , Y j [ t − τ ] , . . .
, Y j [ t − kτ ]) .
Now, if
X unidirectionally causes Y ( i.e. , Y does not cause X), it means that X is part of an autonomous dynamical system and that Y is part of a larger one, containing X. The attractor of Y is then the same as the one of the joint dynamical system Z .
By contrast, the attractor of X is only a subset of it.
From Taken’s theorem, it is theoretically possible to recover the original M Z from M (cid:48)
Y and hence, by extension, recover M (cid:48)
X from M (cid:48)
Y .
However, the contrary is not true and it is in general not possible to recover M (cid:48)
Y from M (cid:48)
X .
The CCM algorithm uses this property to infer causal dependency.
It embeds both dynamical systems X and Y and use k -nearest neighbors to predict points on M (cid:48)
X from M (cid:48)
Y and inversely.
The result then consists in the correlation of the predictions with the true values.
We write C cm( X, Y ) the Pearson correlation for the task of reconstructing M (cid:48)
X from M (cid:48)
Y ,
[Equation]
where ˆ M (cid:48)
X stands fro the prediction of M (cid:48)
X obtained from M (cid:48)
Y .
Importantly, this measure is nonsymmetric as an non-injective map between M (cid:48)
X and M (cid:48)
Y would lead to an accurate reconstruction being possible in one direction only.
To infer that there is a causal link between the predictor dynamical system and the predicted one, this correlation should be high and, importantly, increase with the length of the observed time series, as the observed manifolds become denser.
The potential results are then interpreted in the following way (1) X causes Y if one can reconstruct with high accuracy M (cid:48)
X from M (cid:48)
Y ; (2) X and Y are not causally related (but not necessarily statistically independent) if nor M (cid:48)
X nor M (cid:48)
Y can be reconstructed from the other; (3) X and Y are in a circular causal relation if both M (cid:48)
Y and M (cid:48)
X can be reconstructed from the other.
In the extreme case of strong coupling, the two systems are said to be in synchrony, and it becomes hard to distinguish between unidirectional or bidirectional coupling (Ye et al., 2015).
3. N EURAL ODE S
Many continuous-time deterministic dynamical systems are usefully described as ODEs.
But in general, not all dimensions of the dynamical system will be observed so that the system is better described as an ODE on a continuous latent process H ( t ) , conditioned on which the observations
X [ t ] are generated.
For instance, when observing only one dimension of a 2 dimensional dynamical system, we cannot ﬁnd a ﬂow φ t ( X ) on that single dimension variable, but we can ﬁnd one on the latent process H ( t ) .
We then have the following description of the dynamics:
[Equation]
where θ represents the parameters of the ODE, f θ ( · ) is a uniformly Lipschitz continuous function and g ( · ) is a continuous function.
Learning the dynamics of the system then consists in learning those parameters θ from a ﬁnite set of (potentially noisy) observations of the process X .
Neural
ODEs (Chen et al., 2018) parametrize this function by a neural network.
Learning the weights of this network can be done using the adjoint method or by simply backpropagating through the numerical integrator.
Note that one usually allows X[t] to be stochastic ( e.g. observation noise).
In that case, the mean of X [ t ] (rather than X [ t ] itself) follows Equation 3.
3. C AUSAL INFERENCE WITH L ATENT CCM
A key step in the CCM methodology is to compute the delay embedding of both time series: Φ( X [ t ]) and Φ( Y [ t ]) .
However, when the data is only sporadically observed at irregular intervals, the probability of observing the delayed samples
X i [ t ] , X i [ t − τ ] , . . .
, X i [ t − kτ ] is vanishing for any t .
X (cid:48)
[ t ] and Y (cid:48)
[ t ] are then never fully observed (in fact, only one dimension is observed) and nearest neighbor prediction cannot be performed.
What is more, short time series usually do not allow to compute a delay embedding of sufﬁcient dimension ( k ) and lag ( τ ) (Clark et al., 2015).
Instead of computing delay embeddings, we learn the dynamics of the process with a continuoustime hidden process parametrized by a Neural ODE (as in Eq. 3) and use this hidden representation as a complete representation of the state-space, therefore eliminating the need for delay-embedding that was limiting the applicability of CCM to long, constant sampling time series.
A graphical representation of the method is shown on Figure 1.
To infer causality between temporal variables from their time series X [ t ] and Y [ t ] , the ﬁrst step is to train two GRU-ODE-Bayes models (De Brouwer et al., 2019), a ﬁltering technique that extends
Neural-ODEs.
Being a ﬁltering approach, GRU-ODE-Bayes ensures no leakage of future information backward in time, an important requirement for our notion of causality.
The continuity of the latent process is also important as it provides more coverage of the attractor of the dynamical system.
Indeed, a constant latent process in between observations (such as obtained with a classical recurrent neural network such as GRU) would lead to fewer unique latent process observations.
The same model is used for all segments of each time series and is trained to minimize forecasting error.
We learn the observation function g , the ODE f θ and the continuous-time latent process H ( t ) .
We write the resulting space of latent vectors from time series X on all segments as H X .
Causality is then inferred by checking the existence of a continuous map between H X and H Y .
Analogously to CCM, we consider X causes Y if there exists a continuous map between H Y and
H X .
This is consistent because H X , just as the delay embedding Φ X , is a embedding of the strange attractor of the dynamical system as stated in Lemma 3.1 for which we give the proof in the Appendix E.
Lemma 3.1.
For a sporadic time series X [ t ] ∈
X satisfying the following dynamics,
[Equation]
with g ( · ) and f θ continuous functions.
If there exists one observation function α H ∈ C 2 : X
→ R along with a valid couple ( k, τ ) (in the Takens’ embedding theorem sense) such that the map Φ k,τg ( φ H ) ,α H ( H ( t )) is injective, the latent process H ( t ) is an embedding of the strange attractor of the full dynamical system containing X .
The requirement of Φ k,τg ( φ H ) ,α H being injective is not enforced in our architecture.
However, with sufﬁcient regularization of the network, it is satisﬁed in practice as shown by our results in Section 4.5.
The same reasoning as in CCM then applies to the latent process and causal direction can be inferred.
The existence of a continuous map between the latent spaces of both time series is quantitatively assessed with the correlation between the true latents of the driven time series and the reconstructions obtained with a k -nearest neighbors model on the latents of the driven time series.
For instance, for a direction X
→ Y , we report the correlation between predictions of H X obtained from H Y and the actual ones ( H X ).
A strong positive correlation suggests an accurate reconstruction and thus a causal link in the studied direction between the variables ( e.g. , X → Y ).
By contrast, a weak correlation suggests no causal link in that direction.
E XPERIMENTS
We evaluate the performance of our approach on data sets from physical and neurophysiology models, namely a double pendulum and neurons activity data.
We show that our method detects the right causal topology in all cases, outperforming multi-spatial CCM, as well as baselines designed to address the sporadicity of the time series.
The code is available in the supplementary material.
4. B ASELINE METHODS
To the best of our knowledge, this is the ﬁrst time CCM is applied to short sporadic time series.
Indeed, because of missing variables, many standard approaches are simply not applicable.
The main baseline consists in multi-spatial CCM (Clark et al., 2015) applied to regularly sampled data with a sampling rate similar to the one of the sporadic data.
We also compare our approach to variants where multi-spatial CCM is applied to an interpolation of the sporadic time series using (1) linear interpolation and (2) univariate and multivariate Gaussian Processes (GP and MVGP).
For the
Gaussian Process, we chose a mixture of RBF and identity kernel and learn the parameters from the data.
To model multivariate GPs (MVGP), we used the combination of a Matern and a periodic
Matern kernel for the time dimension and used co-regionalization (Bonilla et al., 2008) with a full-rank interaction matrix.
We then use the mean of the posterior process as the reconstruction subsequentially fed to the classical CCM method.
Implementation was done with GPﬂow (Matthews et al., 2017).
We also compared our approach to non-CCM causal discovery methods such as PCMCI (Runge et al., 2019) and VARLinGAM (Hyv ¨ arinen et al., 2010).
PCMCI uses conditional independence testing between time series at differrent lags to infer causal dependencies.
VARLinGAM learns a graphical model of the longitudinal variables and their time lags, using the LinGAM method (Shimizu et al., 2006).
These methods do not allow for short sporadic time series as input but a comparison with a less challenging non-sporadic variant of our datasets is presented in Appendix F.
4. P ERFORMANCE METRICS
Our method assesses causality by detecting convergent reconstruction accuracy between the latent processes of different time series.
To account for both aspects in a single score, we use the difference between the correlation of the reconstruction and the target latent vector using the whole data ( C cm full ) and the correlation using only 100 sample points ( C cm 0 ), as shown on Figure 2 and suggested in Clark et al.
The score for the causal coupling from X [ t ] to Y [ t ] is then deﬁned as
[Equation]
with a higher score implying more conﬁdence in a causal relationship.
Additionally, to quantify the certainty about the presence of a causal edge in the data generation graph, we compare the obtained scores with the ones that would be obtained with CCM on fully observed but independent time series.
We compute the Mann-Whitney U -statistics (Ma et al., 2014) and provide the corresponding p -value.
[Figure]
e c o n s t r u c t
i
o
n c
[Figure]
rr
[Figure]
l a t i o n
[Figure]
Nevertheless, in practice, one might not have access to the score of independent time series, making it difﬁcult to assess from the score only if a causal relationship is present.
To address this issue we visualize the results graphically as shown in Figure 2.
Causal directions should then stand out clearly and have the characteristic convergent pattern (Sugihara et al., 2012)
4. D OUBLE PENDULUM
Description.
The double pendulum is a simple physical system that is chaotic and exhibits rich dynamical behavior.
It consists of two point masses m 1 and m 2 connected to a pivot point and to each other by weightless rods of length l 1 and l 2 , as shown on Figure 4 in
Appendix A.
The trajectories of the double pendulum are described by the time series of the angles of the rods with respect to the vertical ( θ 1 and θ 2 ), as well as the angular momenta p 1 and p 2 conjugate to these angles.
Each trajectory is then a collection of 4-dimensional vector observations.
To introduce causal dependencies from pendulum X to Y , we include a non-physical asymmetrical coupling term in the update of the momentum conjugate to the ﬁrst angle:
[Equation]
where c X,Y is a coupling parameter.
The term corresponding to a quadratic potential incorporated to the Hamiltonian of system Y results in an attraction on system Y by system X .
Depending on the values of c X,Y and c Y,X , we have different causal relationships between X and Y .
Namely, (1)
X causes Y iff c X,Y (cid:54)
= 0 , (2) Y causes X iff c Y,X (cid:54) =
and (3) X is not causally related to Y if c X,Y =
c Y,X = 0 .
Data generation .
We consider two cases of generating models.
The ﬁrst one consists of two double pendulums ( X [ t ] and Y [ t ] ) with high observation noise with Y causing X .
In this case, we set c X,Y = 0 and c Y,X = 0 .
The second consists of 3 double pendulums ( X [ t ] , Y [ t ] and Z [ t ] ), with one of them causing the other two ( c Z,X = 0 .
5 , c Z,Y = 1 ).
We then infer the causal model relations between those 3 variables in a pairwise fashion ( i.e. we infer the causal direction between all pairs of variables in the system).
Remarkably, X and Y are here correlated but not causally related.
Graphical representations of both considered cases are presented in Figure 3.
Parameters of the pendulums (lengths and masses) are presented in Appendix A.
We generate 5 trajectories with different initial conditions ( θ 1 ∼ N ( − 1 , 0 . 05) and θ 2 ∼ N (0 . 5 , 0 . 05) ).
We simulate observation noise by adding a random Gaussian noise n to the samples with n ∼ N ( µ = 0 , σ = 0 . 1) for the ﬁrst case and n ∼ N ( µ = 0 , σ = 0 . 01) for the second.
To account for the short length of time series usually encountered in the real world, we randomly split the trajectories in windows ofseconds.
To simulate sporadicity, we sample observation uniformly at random with an average rate of 4 samples per second.
Furthermore, for each of those samples, we apply an observation mask that keeps each individual dimension with probability 0 .
This whole procedure leads to a sporadic pattern as shown in Figure 5 of Appendix A. We used 80% of available windows for training and used the remaining 20% for hyperparameter tuning with the MSE prediction on future samples used as model selection criterion.
More details on this procedure is given in Appendix G.
[Figure]
4. N EURAL ACTIVITY DATA
We also evaluate our approach on neural activity data.
We generate time series of the average membrane potential of two populations of leaky integrate-and-ﬁre neurons with alpha-function shaped synaptic currents ( iaf psc alpha ) simulated by NEST-2.20.0 (Fardet et al., 2020).
Each neuron population contains 100 units with sparse random excitatory synapses within the population.
We consider two cases, one where population A unidirectionally excites population B and another case where both populations ﬁre independently.
To account for the short and sporadic nature of real-world data, we generate 5,000 windows of 20 seconds from which we sample 1 observation every second on average.
This leads to 20 samples being available on average per time window.
4. R ESULTS
Results over 5 repeats for the double pendulums and the neural activity data are presented in Tableand in Figure 2.
Because this method uses different metrics for inferring the underlying causal graph, the results for PCMCI are presented in Appendix F, where we show that the method cannot reliably infer the generative causal dependencies in our data.
Double Pendulum .
Our approach is the only one to recover the right causal direction from the sporadic data.
The other baselines do not detect any signiﬁcant correlation and thus no causal link between double pendulums.
Despite having access to constant sampling data, multi-spatial CCM is also not able to detect the right data structure.
We argue this is caused by the short length of time series window, and thus the low number and quality of delay embeddings that can be computed.
In contrast, as our method shares the same model across all time windows, it represents more reliably the (hidden) state-space at any point in time.
Importantly, the perfect reconstruction for Caseshows that we can distinguish confounding from correlation between time series.
Indeed, when inferring causal directions between X and Y , variable Z is not used and thus hidden .
Yet, our methods detects no causal relation between X and Y .
Figure 2(a) graphically presents the results of our method for the second case, where it is obvious that the only two convergent mappings are the ones corresponding to the true directions (solid blue and green lines), providing a strong signal for the right underlying causal mechanism.
Neural activity For the neuron activity data, we observe that our method delivers the largest effect size towards the true data generating model ( S c = 0 . 295 ).
Baselines methods relying on imputation do not provide any clear signal for a causal coupling (score 10 times lower).
Multi-spatial CCM with the regularly sampled original data provides similar signal than our approach but dampened.
Interestingly, we observe a small but signiﬁcant correlation in the wrong direction ( A ← B ) suggesting a small coupling in this direction.
An inspection of Figure 2(b), however, will convince the reader
[Table]
that the main causal effect is indeed from A to B .
This small correlation in the direction A ← B is also observed in the fully observed data as shown in Figure 6 in Appendix B
[Table]
C ONCLUSION AND FUTURE WORK
In this work
[Table]
we propose a novel way to detect causal structure linking chaotic dynamical systems that are sporadically observed using reconstruction of underlying latent processes learnt with
Neural-ODE models.
We show that our method correctly detects the causal directions between temporal variables in a low and irregular sampling regime, when time series are observed in only short noncontiguous time windows and even in the case of hidden confounders, which are characteristics of real-world data.
Despite the apparent limitation of our method to chaotic systems, it has been shown that CCM is broadly applicable in practice as many real dynamical systems are either chaotic or empirically allow Takens’-like embeddings.
As our work builds upon CCM theoretically, we expect the range of application to be at least as large and leave the application to other real-world data for future work.
A D OUBLE PENDULUM
Figure 4 presents a graphical representation of a double pendulum with its two masses and two weightless rods.
Figure 5 shows examples of trajectories generated by a double pendulum.
The double pendulum is a simple physically system that is chaotic and exhibits rich dynamical behavior.
The Lagrangian of the double pendulum is
[Equation]
The corresponding Hamiltonian can be derived using Legendre transform H
= (cid:80) i ˙
θ
i p i − L .
The system evolution can be simulated by integrating the Hamilton equations:
[Equation]
The Jacobian of the right hand side is
[Equation]
Note that the diagonal elements cancelling in pairs, which results in a trace of zero that indicates the volume-preserving property of the Hamiltonian ﬂow according to Liouville’s theorem.
This property corresponds to information preservation in nondissipating physical systems.
Consequently, a noncoupled double pendulum does not have a proper attractor.
However, for a given initial condition, and thus given energy, the possible states still form a densely populated volume in state-space.
Applying the nonphysical coupling term, the conservation rule do not hold anymore.
The real part of the eigenvalues of J are called the local Lyapunov exponents.
The direction of the largest expansion evolves as
[Equation]
The largest Lyapunov exponent is given by
[Equation]
Note that in stationary processes J is constant, and the differential equation have a closed form solution
[Equation]
and the local and global Lyapunov exponents are equal.
The largest Lyapunov exponent can be described intuitively as
[Equation]
where δ ( t ) is deﬁned as the difference between two phase-space trajectories, with initial condition inﬁnitesimally close to each other:
[Equation]
We use numerical integration to compute the largest Lyapunov exponent of the double pendulum, and verify that it is in the chaotic regime.
B I NTERACTING NEURON POPULATIONS
The time series is the average membrane potential of two populations of leaky integrate-and-ﬁre neurons with alpha-function shaped synaptic currents ( iaf psc alpha ) simulated by NEST-2.20.0 (Fardet
[Table]
et al., 2020).
Each population contains 100 units with sparse random excitatory synapses inside population
[Table]
and unidirectionally from population A to population B. A Poisson generator with rate of 40kHz was used to excite the network.
[Table] [Table]
In Figure 6, we plot the reconstruction correlations of the coupled neuron populations obtained with the fully observed time series (10 observations per second) and evaluated with standard CCM.
We observe a small convergence in the reconstruction in the non-causal direction, indicating potential synchrony.
As presented in Table 1, our approach also captures this small reconstruction signal.
C R ESULTS WITH UNIVARIATE G AUSSIAN P ROCESSES
In Table 5, we present the results of our experiments with univariate Gaussian Processes (GP).
In this case, we only learn a GP on the dimension of interest to compute the delay embeddings.
As we can see, results are slightly worse than when using multivariate Gaussian Processes (MVGP).
[Figure]
D ATA
[Table]
D F AILURE OF THE G RANGER CAUSALITY FRAMEWORK
To show how the Granger causality framework fails in the general nonlinear dynamical systems case, we consider the following coupled dynamical system:
[Equation]
Following Granger causality, including values of Y for predicting X [ t + 1] should increase the prediction accuracy, and thus hint towards a causal effect of Y on X .
However, dynamics of X [ t ] can be rearranged such that all information about Y [ t ] is contained in X
[ t ] already.
Indeed,
[Equation]
Conditioning on Y [ t ] would not bring additional information and Granger causality would then fail to uncover the right causal structure.
E P ROOF OF L EMMA 3.
Proof.
We ﬁrst write the map Φ kg ( φ H ) ,α H ( H ( t )) in its full form: Φ kg ( φ H ) ,α H ( H ( t )) :
H → R k s.t.
Φ
[Equation] [Equation]
( t ))))
[Equation] [Equation]
t )))
[Equation]
where the last line follows from the deﬁnition of the dynamical system.
Φ then maps the latent process to the delay embedding of X obtained with observation function α H .
As the observation function α H ∈ C 2 , the ﬂow φ H and the function g ( · ) are all continuous, this implies that the map Φ is also continuous in H ( t ) .
It is also surjective as all delay embeddings (or points in the state-space) will have at least one latent value generating this delay embedding.
Indeed, if we write M (cid:48)
α H as the shadow manifold of the delay embeddings of X with observation function α H , we have that
[Equation]
Let us now assume that there exists a speciﬁc observation function α ∗ H such that Φ is injective.
The map Φ α ∗ H is then bijective.
Furthermore, as both H and M (cid:48)
α ∗ H are endowed with a metric, the map Φ α ∗ H is a homeomorphism between H and M (cid:48)
α ∗ H .
We now show that Φ is a homeomorphism for any observation function.
From Takens’ theorem, any delay embedding with valid observation function α , dimension k , and delay τ is a valid embedding of the strange attractor of the dynamical system.
There must then exists a homeomorphic map Ψ between any two valid delay embeddings with different observation functions:
[Equation]
By transitivity, there is now a homeorphism between H and any valid delay embedding deﬁned with observation function α H deﬁned as Ψ α ∗ H ,α H ◦ Φ kg
( φ H ) ,α ∗ H ( H ( t )) .
By Takens’ theorem, H is thus an embedding of the strange attractor of the dynamical system containing X [ t ] .
F C OMPARISON WITH PCMCI AND VARL IN GAM F. PCMCI
We compared our approach with PCMCI (Runge et al., 2019), a recently introduced method to estimate causal networks from large-scale time series datasets.
The method uses independence tests at various time lags to infer causal links between time series.
The method does not allow for sporadic time series as a constant time lag is required for the conditional independence tests.
Furthermore, the method does not support a way to handle a collection of short time series from a common dynamical system.
We then used the method on a less challenging variant of our data where the observations are sampled at a constant rate corresponding to the sampling rate used for generating the sporadic time series.
We only feed a very long time series without interruption.
We used a maximum time lag of 10 seconds and report the results of cases 1 and 2 of the double pendulum for various signiﬁcance thresholds in Table 6.
We used the implementation of the method provided by the authors at https://github.com/jakobrunge/tigramite/ .
Because the method infers causality with a different score than ours, we report the inferred conﬁguration at each repeat and for each case.
The different conﬁgurations are shown on Figure 7.
For case 1, we observe that
PCMCI recovers the true causal graph 2 times out of 5 when the signiﬁcance is set to p < 0 .
For case 2, at all levels of conﬁdence, PCMCI infers a fully connected graph (conﬁguration ∆ ).
We suggest this results form the large number of time series for this conﬁguration ( 12 ) as well as the existence of complex coupling dynamics of the chaotic dynamical system, make the causal inference challenging.
An example of inferred causal network with PCMCI (case 1 with signiﬁcance level of p < 0 .
001 ) is presented on Figure 8.
F. VARL IN GAM
We also compared the Latent CCM method with VARLinGAM
(Hyv ¨ arinen et al., 2010).
VARLinGAM detects causal links between longitudinal variables by learning a directed acyclic graph of interactions of the variables and their time lags.
In particular, VARLinGAM derives the best acyclic graph with the LinGAM method (Shimizu et al., 2006).
We infer causality between the group of variables X and another Y by checking the existence of causal edges between individual variables of X and Y .
We used the implementation of VARLinGAM available at https: //github.com/
cdt15/lingam and use a maximum time lag of 10 seconds (same as for
PCMCI) and a minimum causal weight of 0.01.
As for PCMCI, the score provided to infer a graph is different than ours and we provide the results of the learnt causal graphs for cases 1 and 2 of the double pendulum in Table 6.
The method only recovers the true graph of case 1 in 60% of the time.
For case 2, the method fails to recover the causal generative model for all the repeats.
Details of the conﬁguration codes used are given on
[Table]
A (
[Table] [Table]
B ( (cid:55) )
Γ ( (cid:51) )
B ( (cid:55) ) B ( (cid:55) )
Γ ( (cid:51) )
[Table]
∆ ( (cid:55) ) ∆ ( (cid:55) )
∆ ( (cid:55) )
∆ ( (cid:55) ) ∆ ( (cid:55) )
p < 0 .∆
( (cid:55) )
∆ ( (cid:55) ) ∆ ( (cid:55) )
∆ ( (cid:55) ) ∆ ( (cid:55) )
V AR L IN GAM
C ASEp < 0 .
A ( (cid:55) ) A ( (cid:55) )
A ( (cid:55) ) A ( (cid:55) )
A ( (cid:55) ) p < 0 .
A ( (cid:55) ) A ( (cid:55) )
Γ ( (cid:51) )
Γ ( (cid:51) ) Γ ( (cid:51) )
C ASEp < 0 .
A ( (cid:55) )
B ( (cid:55) ) B ( (cid:55) )
B ( (cid:55) )
Γ ( (cid:55) ) p < 0 .Γ
( (cid:55) )
Γ ( (cid:55) ) A ( (cid:55) )
Γ ( (cid:55) ) Γ ( (cid:55) )
[Figure]
A conﬁguration
X
Y
B conﬁguration
X
Y Γ conﬁguration (a) Case
X
Y
Z
A conﬁguration
X
Y
Z
B conﬁguration
X
Y
Z Γ conﬁguration
X
Y
Z ∆ conﬁguration
[Figure]
G M ODEL SELECTION PROCEDURE
When training GRU-ODE-Bayes models, we use 80% of the available time series samples as training set and the remaining 20% as validation set.
For training, we use the reconstruction loss proposed by the authors (De Brouwer et al., 2019).
For validation, we feed the time series until half of the available horizons ( e.g. for time series of length 10 seconds, we feed the 5 ﬁrst seconds) and compute the MSE on the reconstruction of the subsequent available samples.
We choose the model hyperparameters that minimize the MSE over the validation set.
Note that we do not need test set as our ultimate goal resides in causal direction inference and not in accurate forecasting of the time series.
Importantly, models for each time series are learnt independently and no information about causal direction is available at any time in the process.
H S PORADIC DATA WITH MISSINGNESS NOT AT RANDOM
Experiments and results presented in Section 4.5 consider a a random sporadic sampling of the data.
The data is thus missing at random (MAR).
In practice, however, the sampling of a process is usually not fully random but rather depends on the value of the process itself.
As a simple example, doctors measure the blood pressure of patients more often when it’s high or likely to be high.
The sampling pattern then gives information about the value of the process we want to model.
In order to account for this bias occurring in practice, we consider also a variant of the double pendulum dataset (case 1) where the missingness is not at random (MNAR) and with noise standard deviation of 0 .
The sampling pattern we consider is the following.
If the absolute value of the angle of the ﬁrst rod θ 1 is larger than π 4 , we sample the process with a probability two times larger than if the angle is smaller than π 4 .
The sampling probability of an observation p s ( X t ) is then :
[Equation]
[Figure]
Note that the total number of observations is still kept constant with respect to the MAR case.
Figure 9 shows the results of latent CCM on the MNAR double pendulum data (case 1).
We observe that latent CCM is still inferring the correct causal directions, despite the sampling bias.
[Figure] [Table]
I U SING A RNN INSTEAD OF A N EURAL -ODE
As Equation 3 in the paper suggests, we consider our observations are generated from a continuous latent process H ( t ) .
Different techniques could be used to infer this process H ( t ) in our latent CCM approach.
Among those techniques, neural-ODE models such as [1] or [2] embody the assumptions of Equation 3 and are thus a natural choice for the inference of informative latent vectors.
Another choice could be to learn those dynamics with a non-continuous recurrent neural network approach.
In this section, we compare Neural-ODE methods with using a standard recurrent neural network (GRU) for learning the dynamics of the process.
Because the processing of missing data across dimensions is not well deﬁned for GRU, we use a version of the data as in Appendix F, namely the observations are sampled at a constant rate (similar to the one used for sporadic data, taking into account the sampling across dimensions) and importantly, no missing dimensions are allowed.
We then use the learnt latent process to infer causal direction for case 2 of the double pendulum data.
Results are presented in Table 8.
We observe that incorrect causal directions are inferred (from X to Z and from Y to Z ).
Some theoretical properties of Neural-ODEs can help explain this result.
Because of their continuous resolution, Neural-ODEs allow to have a denser coverage of the attractor we want to reconstruct.
This feature is further strengthened by the fact that different integrators can be used to recover the latent process, therefore allowing to tune the resolution of the learnt latent process.
In the case of physical systems, a symplectic integrator can also be used, to ensure conservation of energy and more accurate learning of the dynamics.
[Table]
J C OMPLEXITY OF THE METHOD AND BASELINES
As all methods require a k-nearest neighbors step for each pair of time series, the difference in computation arises in the computation of the embeddings fed to the kNN.
We then report the complexity of computing the embeddings to be used for the cross-mapping in Table 9.
Computing delay embeddings scales linearly in the number of embedding dimensions ( H ) and the number of samples in each time series ( M ).
When using Gaussian Processes, one has ﬁrst to infer the latent process at all time points and invert a covariance matrix of size D × D which requires an additional O ( D 3 × M ) .
For latent-CCM, we avoid the computation of the delay embeddings but we require to train a NeuralODE model which requires O ( H 2 ) for each time step and at each observation.
[Table]