Cause-effect inference through spectral independence in linear dynamical systems: theoretical foundations.
BESSERVE
DE
NAJIS @
CMU
JANZIND @
AMAZON
DE
BS @ TUE .
MPG .
DE
DJ contributed before joining Amazon.
Editors: Bernhard Sch ¨ olkopf, Caroline Uhler and Kun Zhang
Abstract
Distinguishing between cause and effect using time series observational data is a major challenge in many scientific fields.
A new perspective has been provided based on the principle of Independence of Causal Mechanisms (ICM), leading to the Spectral Independence Criterion (SIC) for time series causally unidirectionally linked by a linear time-invariant relation.
SIC postulates that the power spectral density (PSD) of the cause time series is uncorrelated with the squared modulus of the frequency response of the filter generating the effect.
Since SIC rests on methods and assumptions in stark contrast with most causal discovery methods for time series, it raises questions regarding what theoretical grounds justify its use.
In this paper, we provide answers covering several key aspects.
After providing an information theoretic interpretation of SIC, we present an identifiability result that sheds light on the context for which this approach is expected to perform well.
We further demonstrate the robustness of SIC to downsampling – an obstacle that can spoil Grangerbased inference.
Finally, an invariance perspective allows to explore the limitations of the spectral independence assumption and how to generalize it.
Overall, these results provide insights on how the ICM principle can be assessed mathematically to infer direction of causation in empirical time series.
of Measure, Cause-effect Pairs
1. Introduction
One purpose of causal inference is to estimate the direction of cause-effect relationships between different parts of a system, to provide insights about the underlying mechanisms and how to intervene on them to influence the overall behavior.
Since interventions are often challenging, or unethical to perform, a number of causal inference techniques have been developed to infer the causal relationships from observational data only (Spirtes, 2010; Pearl, 2000).
For this aim, they rely on key assumptions pertaining to the mechanisms generating the observed data.
Classical constraint-based search methods, such as the PC algorithm (Spirtes et al., 1993), address this question by assuming successive observations are independent samples from the same unmanipulated density in order to characterize it and infer compatible causal graphical models.
However, these methods cannot infer the causal direction when the graphical model consists only of two variables.
Moreover, observed data from complex natural system are often not i.i.d.
and time dependent information reflect key aspects of these systems.
On the other hand, causal relations between time series are typically explored via Granger-causality type methods (Granger, 1969), which require causal sufficiency.
To overcome this limitation, more recent approaches (Mastakouri et al., 2021) (and references therein) employ Markov condition and causal faithfulness to identify characteristic patterns of conditional (in)dependences that witness causal influence even in the presence of hidden common causes.
Most common practical implementations of Granger causality rely on assumptions involving vector autoregressive structural equations and exogenous i.i.d.
random variables called the innovations of the process (Granger, 1969; Peters et al., 2013).
These methods can successfully estimate causal relationships when empirical data is generated according to the assumptions, but the results can be misleading when the model is misspecified.
In particular, Granger causality may fail to infer the true direction of causation when the sampling of the time series is not fast enough to capture the dynamical interactions precisely (Geweke, 1982;
Gong et al., 2015), an issue that also spoils approaches like (Mastakouri et al., 2021), which also –like Granger– relies on observations of ground truth dynamics at well-defined points in time.
A different approach for inferring causal directions in time series, the Spectral Independence
Criterion (SIC), has been introduced in (Shajarisales et al., 2015).
In contrast to the above mentioned Granger-like methods, SIC relies on the philosophical principle that the mechanism that generates the observed cause variable and the mechanism that generates the effect from the cause are chosen independently by Nature, such that these two mechanisms do not inform each other.
Possible formalizations of this postulate of Independence of Causal Mechanisms (ICM), have been proposed in (Janzing and Sch¨olkopf, 2010; Lemeire and Janzing, 2012) (where ‘independence’ amounts to algorithmic independence) and (Sch¨olkopf et al., 2012) (where ‘independence’ amounts to semisupervised learning being useless).
These abstract and general ideas have been further exploited to design several concrete domain-specific causal inference methods (Daniusis et al., 2010;
Janzing et al., 2012, 2010a; Zscheischler et al., 2011; Sgouritsa et al., 2015; Shajarisales et al., 2015) among which SIC was the first to address the case of time series.
It introduces an equation to formulate the
ICM postulate when both the cause and the effect are stationary time series and the cause generates the effect trough a linear time invariant mechanism.
This framework leads to defining a quantity called Spectral Density Ratio (SDR), quantifying to which extent the ICM postulate is approximately satisfied.
The SDR is exploited to decide the direction of causation when a pair of time series is considered, notably in the context of neural times series (Shajarisales et al., 2015; RamirezVillegas et al., 2021), as well as to assess the ICM and extrapolation capabilities of convolutional generative models (Besserve et al., 2021).
Despite these works, the theoretical underpinnings of
SIC remain largely unexplored.
The present paper aims at establishing identifiability results as well as connections to other existing frameworks in causality.
After introducing the basic definitions of our causal inference framework (Sec. 2), we introduce the SIC assumption in Sec. 3.
We then provide an information geometric perspective on SIC (Sec. 4), as well as theoretical guaranties for identifiability of the causal direction (Sec. 5) that are robust to time-series downsampling (Sec. 6).
Finally, an invariance perspective on SIC allows to define generalizations of SIC adapted to specific application domains (Sec. 7).
All proofs are provided in Appendix A. Overall, these result clarify the conditions under which SIC is applicable, its limitations and potential extensions.
2. Background and Model description 2.1. Deterministic cause-effect inference setting
Cause-effect inference based on the ICM principle can be explained as follows.
We assume we have two observables X and Y , possibly multidimensional and neither necessarily from a vector space, nor necessarily random.
We assume additionally that these two observations have a deterministic, acyclic, unconfounded relation: they can be mapped to each other via some invertible map f , and their generative model obeys one of the following structural equations:
[Equation]
In the remainder of the paper, we will assume that exactly one of these causal models is the true one (that means we exclude the possibility of a confounder, for example).
Without loss of generality,
X → Y is considered to be the true generative model, called the forward causal model , while Y →
X will be called the backward model .
We will call f the mechanism, X the cause and Y the effect.
Identifying that the forward model is the true one from observations of X and Y only, is challenging since both models provide a satisfactory description of the data.
As a consequence, an additional assumption needs to be added to this setting to inference the direction of causation.
The ICM framework relies on the eponymous postulate that properties of X and f are “independent” in some sense.
ICM-based methods explore specific settings for which, if the assumption is valid for the true generative model, the converse independence assumption (between f − 1 and Y ) is very unlikely.
As a consequence, the true causal direction can be identified by evaluating this independence in both scenarios and pick the direction that is closer to satisfying the idealized independence assumption.
The remainder of this section will introduce an ICM setting which is specific to time series.
2.2. Linear filters
We assume that the ground truth causal mechanism which links a cause time series x to an effect time series y , is a (deterministic) linear time invariant filter, such that
[Equation]
) where h denotes the impulse response of the filter and ∗ denotes discrete time convolution.
Note that this assumption implies that the causal relation is unconfounded.
We assume that the filter satisfies the Bounded Input Bounded Output (BIBO) stability property (Proakis, 2001), stating that any bounded input x (such that sup t | x t | < + ∞ ) results in a bounded output.
In our setting, a necessary and sufficient condition for BIBO stability is h x → y ∈ ℓ 1 ( Z ) , i.e. (cid:80) t | h x
→ y ,t | < + ∞ .
A filter h is called causal whenever h τ = 0 for τ < 0 and Finite Impulse Response (FIR) when the transfer function has a finite support.
2.3. Stationary sequences
We assume that the input time series x is a sample drawn from a real-valued zero mean weakly stationary process (Brockwell and Davis, 2009), { X t , t ∈ Z } , and denote by C xx ( τ ) =
E [ X t X t
+ τ ] the autocovariance function of the process, which does not depend on t due to stationarity.
We also assume that (cid:80)
τ ∈ Z | C xx ( τ ) | < + ∞ such that its Power Spectral Density (PSD) S xx , defined as the Discrete Time Fourier Transform of the autocovariance function, is a well defined 1 -periodic function of normalized frequency ν ∈ R .
Background on the DTFT can be found in Appendix B.1.
Under these assumptions, S xx is bounded, continuous, and its average over the unit length intervale
I =
[ − 12 , 12 ) corresponds to the power of the process P ( X ) =
E ( | X t | 2 ) =
⟨ S xx ⟩ which is thus well-defined and finite.
To simplify notations, we will denote by ⟨ .
⟩ the integral (or average) of a function over the unit length interval I , such that P ( X ) = ⟨ S xx ⟩ .
When such a sequence is fed into a filter of impulse response
h X
→ Y as defined above, the stochastic output Y is weakly stationary with summable autocovariance such that
[Equation]
where (cid:98)
h X
→ Y denotes the DTFT of the impulse response (see Appendix B.1 for details), called the frequency response of the system.
This follows from elementary properties of the Fourier transform and Proposition 3.1.2.
in (Brockwell and Davis, 2009).
If such a BIBO-stable filtering relationship exists in only one direction (i.e. when the frequency response is not invertible at some frequency), it is natural to assign causality to the stable direction (given the Cause-Effect Inference setting described in section 2.1).
If BIBO-stable filters and thus impulse responses can be defined for both directions, we can denote them with h X → Y and h Y → X , respectively, and their Fourier transforms are continuous and linked by
[Equation]
In such a situation, both the forward and backward filtering models are plausible structural causal models to explain the observed time series.
Therefore a more sophisticated criterion is needed for causal inference.
We focus on this challenging situation in the present work and will require basic assumptions for the causal models in the remainder of the paper, summarized as follows.
Assumption 1 (Invertible causal model)
The cause X is a weakly stationary time series with
C xx ∈ ℓ 1 ( Z ) and such that S xx is strictly positive at all frequencies.
The mechanism is an invertible
BIBO-stable filter with impulse response
h X
→ Y such that its inverse is also BIBO-stable.
The effect is defined as
[Equation]
3. Spectral Independence Criterion (SIC) 3.1. Definition
Given the two stationary processes X := { X t : t ∈ Z } and Y :=
{ Y t : t ∈ Z } such that X causes Y through a linear filter, the ICM hypothesis introduced by Shajarisales et al.
(2015) can be stated as:
Postulate 1 (Spectral Independence Criterion (SIC))
A causal model satisfying Assumption 1 satisfies spectral independence whenever
[Equation]
In practice, this postulate is hypothesized to hold only approximately, and we will explain what is meant by that in Section 5.
However, in Secs.
3-4, we will develop theoretical results based on the perfect spectral independence postulate, that is, where eq.
(3) holds exactly.
As eq.
(2) indicates, the filter applies an amplifying factor to the input power at each frequency to provide the output power, and eq.
(3) makes a statement on the average amplification achieved across frequencies.
Indeed, the left hand side of eq.
(3) is the average PSD of the output signal { Y t , t ∈ Z } over all frequencies, i.e. its total power P ( Y ) .
Hence, SIC states that the output power can be computed by applying the frequency-averaged amplifying factor to the input power.
This suggests that the amplification implemented by the mechanism is not informed by the values of the input PSD at each frequency.
Note that based on (2) and under Assumption 1, the postulate of eq.
(3) can be rephrased using the
PSDs of X and Y alone, i.e.
[Equation]
3.2. Measuring spectral dependence
Shajarisales et al.
(2015) then introduce the scale invariant quantity
ρ X
→ Y measuring the departure from the SIC assumption, i.e. the dependence between input power spectrum and the frequency response of the filter: the Spectral Dependency Ratio (SDR) from X to Y is, under Assumption 1, given by
[Equation] [Equation]
Moreover ρ X → Y can be written in terms of total signal powers and energy of the impulse response:
[Equation]
To interpret (6), note that the filter h X
→ Y amplifies the power of a white noise, i.e. a stationary stochastic process whose power spectrum is constant, by a factor ∥ h X → Y ∥ 22 .
Thus, the SDR measures how much the filter amplifies the power of the cause signal compared to the case in which the cause was a white noise.
We then define ρ Y → X by exchanging the roles of X and Y in the above equations.
We provide a probabilistic interpretation of spectral independence in Appendix B.2 and an intuitive example illustrating its meaning in Appendix B.3.
4. Information geometric interpretation
While Shajarisales et al.
(2015) elaborated on the connection of SIC with the Trace Method (Janzing et al., 2010b), we now investigate its relation to another type of ICM-based approch: InformationGeometric Causal Inference (IGCI) (Daniusis et al., 2010).
We get inspiration from Janzing et al.
(2012) who established a connection between IGCI for linear relationships and the Trace Method.
At the heart of this derivation lies an information geometric interpretation of the principle of independence of cause and mechanism for probability distributions.
After introducing this view, we will show how SIC can be casted into the same framework, in the context of Gaussian processes.
4.1. Information geometry background
Information Geometry is a discipline where ideas from differential geometry are applied to probability theory.
Probability distributions are represented as points from a Riemannian manifold, known as statistical manifold .
Equipped with Kullback-Leibler divergence as premetric 1 , one can study the geometrical properties of the statistical manifold.
For more on this, see e.g. (Amari and Nagaoka, 2007).
[Figure]
M e c han
i s m
For two probability distributions P, Q , D ( P ∥ Q ) will denote their Kullback-Leibler divergence, also called the relative entropy distance.
Given a deterministic causal structural equation of the form
Y :
= f ( X ) , and given P X and P Y , the distributions of the cause and effect, respectively, we assume the irregularity of each distribution can be quantified by evaluating their divergence to a reference set E of “regular” distributions
[Equation]
We assume moreover that these infima are reached at a unique point, their projection on E
[Equation]
Assuming cause X and effect Y are n -dimensional multivariate Gaussian random vectors linked by a deterministic linear map A , Janzing et al.
(2010b) introduced the ICM-based Trace Condition, that we restate in this restricted setting of equal dimension of cause and effect:
Postulate 2 (Trace Condition) Let tr denote the matrix trace.
The respective covariance matrices of cause and effect, Σ X and Σ Y , satisfy
[Equation]
If we assume E is the set of isotropic Gaussian distributions, the Trace Condition has been shown to be equivalent to:
[Equation]
where → P Y is the distribution of f ( X ) :
= AX when X is distributed according to U X , where U X is the isotropic Gaussian distribution with the same variance as X .
This relation can be interpreted as an orthogonality principle by considering the Kullback-Leibler divergences as a generalization of the squared Euclidean norm for the geodesics P Y U Y , P Y
→ P Y and → P Y U Y as illustrated in Fig.
Since applying the bijection f − 1 preserves the divergences, we get the equivalent relation
[Equation]
This orthogonality principle thus reflects the additivity of irregularities, in the sense
that D ( P Y ∥
U Y ) =
D ( P Y ∥E ) corresponds to the irregularity of Y .
It measures the distance to the set of “regular” distributions, while D ( P X ∥ U X )
=
D ( P X ∥E ) measures the irregularity of X in the same way.
In addition, it can be shown that D ( → P Y ∥ U Y ) =
D ( → P Y ∥E ) and it thus measures the irregularity of the mechanism f indirectly, via the “irregularity” of the distribution resulting from applying f to a regular distribution U Y .
We now show that spectral independence encodes a similar relation for discrete time stationary Gaussian processes.
4.2. Information geometry of stochastic processes
The generalization of KL-divergence to a pair of stationary time series ( X , Y ) is called the relative entropy rate defined as (Ihara, 1993)
[Equation]
where X i : j is the subvector of vector X from element i to element j .
In the case of stationary
Gaussian processes, this quantity can be computed explicitly.
Proposition 1 Let X and ˜X be zero mean weakly stationary discrete Gaussian processes with
PSD’s S xx and S ˜ x
˜ x , respectively, with C xx , C ˜ x ˜ x ∈ ℓ 1 ( Z ) and such that S xx , S ˜ x
˜ x are strictly positive at all frequencies (see also Assumption 1) the relative entropy rate is given by
[Equation]
The proof is a direct application of (Ihara, 1993, Theorem 2.4.4.).
4.3. SIC as an orthogonality principle
We consider ¯ E the manifold of Gaussian white noises: i.e. Gaussian processes with constant PSD, as the set of regular distributions replacing E .
Let P ¯ E X be the information geometric projection of
X on ¯ E .
This is then a Gaussian white noise of power P ( X ) (see Lemma 2).
We then have:
Theorem 1 Let Assumption 1 hold and further assume that X is a Gaussian process.
Set U X =
P ¯ E X , U Y
= P ¯ E Y , and let → P Y be the distribution obtained by feeding a white noise process with distribution U X into S (thus convolving by h X → Y ).
Then we have:
[Equation]
As a consequence the following corollary shows that SIC corresponds to orthogonality of PSD variations in information space, in the case of weakly stationary Gaussian processes.
Corollary 1 Under assumptions of Theorem 1, the SIC postulate is equivalent to the orthogonality of irregularities relative to white-noise Gaussian processes:
[Equation]
5. Identifiability results
The proposal by Shajarisales et al. (2015) to use SIC for inferring the causal direction is essentially based on two insights: (i) an argument for why the SDR is expected to be close to 1 for the causal direction; (ii) an argument for why the SDR is not expected to be close to 1 in the anti-causal direction.
After reviewing these arguments, we show for the first time that they can be exploited to show identifiability of the causal direction based on SDR in the following toy generative model.
5.1. Generative model
Assume a length m Finite Impulse Response (FIR) h ,
such that h τ = 0 for all τ < k and all τ ≥ k + m , for some m and k .
Then h is parametrized by an m -dimensional real vector b such that h
i + k
= b i i = 0 , . . .
, m − 1 .
We assume that b has been generated by nature as a single sample drawn from a random variable B with a spherically symmetric distribution (see for example (Bryc, 2012, Chapter 4) for a rigorous definition).
This implies B takes the form
[Equation]
where R ≥ 0 is a real valued radius random variable, and U is a random vector uniformly distributed on the unit sphere in R m .
U and R are statistically independent, which entails that for any orthogonal transformation T ∈ O ( m ) , the distribution of T b is identical to the distribution of b .
An important family of spherically symmetric distributions is the zero mean isotropic gaussians (i.e. with a covariance matrix that is a multiple of the identity matrix).
The general idea behind this assumption is that the distribution of the impulse response is invariant to a reparameterization of the vector space in a new system of coordinates that preserves the energy of the impulse response.
Theorem 2 shows that for large m the resulting filter will approximately satisfy SIC with high probability.
Theorem 2 (Concentration of measure for FIR filters)
Assume a length m mechanism’s impulse response
h X
→ Y with a spherically symmetric generative model and a fixed input PSD S xx , then for any given ε > 0 , with probability at least 1 − 2 exp( − m 3 ε 2 ) , we have
[Equation]
The exponential term of the probability bound shows that if m is large enough, we can get a tight bound around 1 for the forward SDR with high probability.
Such a concentration of measure result has been provided by Shajarisales et al.
(Theorem 1 therein), however the new bound that we prove here guaranties a much faster concentration when the number of filter coefficients increases due to the term m 3 , instead of m inside the exponential.
5.2. Forward-backward inequality
The following result from (Shajarisales et al., 2015) shows additionally that the dependence measures in both directions are related, as their product can be bounded as a function of the coefficient of variation of | (cid:98)
h X
→ Y | 2 along the frequency axis, defined as the ratio of the standard deviation to the mean
CV ( | (cid:98) h X
→ Y
[Equation] [Equation]
[Figure]
Proposition 2 (Forward-backward inequality)
For a given linear filter S with impulse response
h X
→ Y , input PSD S xx and output PSD S
yy , let S have a non-constant modulus frequency response (cid:98)
h X
→ Y , and assume there exists α > 0 such that for all ν ∈
I ,
[Equation]
Then
[Equation]
Note that α < 1 because a random variable cannot be everywhere smaller than its mean.
According to (11), large fluctuations of | (cid:98) h
X → Y | 2 around its mean guarantee that the product of the independence measures will be bounded away from 1 .
Assuming the SIC postulate is satisfied in the forward direction such that ρ X → Y = 1 , it follows naturally that ρ Y → X < 1 .
5.3. Inference algorithm
Based on the above two steps, Shajarisales et al.
(2015) have introduced a SIC-based algorithm to infer the direction of causation.
First, Theorem 2 guaranties that ρ X
→ Y concentrates around one, such that with high probability 1 − ξ ≤ ρ X → Y ≤ 1 + ξ (the closer is ξ to zero, the tighter is the bound) with ξ > 0 .
Second, Proposition 2 implies a lower bound for the backward SDR based on the foward SDR value.
Indeed, using Proposition 2, for any η > 0 , ρ Y → X ρ X
→ Y < 1 − η leads to
[Equation]
Thus, combining both results, we get ρ Y → X < 1 − η (1 − ξ ) 2 ρ X
→
Y and obtain that ρ Y → X is strictly smaller than ρ X → Y with if we can have 1 − η < (1 − ξ ) 2 .
This qualitative reasoning, illustrated in Fig.
2(b), suggest that a good strategy to infer the direction of causation is to choose the direction with largest
SDR.
The corresponding causal inference algorithm is described in Algorithm 1.
5.4. Identifiability of the generative model
1: procedure SIC I NFERENCE
( X , Y ) 2:
Calculate ρ X → Y and ρ Y →
X using (5) 3: if ρ X → Y > ρ Y
→
X
return
X
→ Y 4: else return Y → X
Proposition 2 argues that SDR is bounded away from 1 for the backward direction, given that the variation of h satisfies some conditions.
This way, we guarantee identifiability of causal directions.
Since we have not explored whether the generating model of Section 5.1 guarantees these conditions to be true, we now show identifiability for this model separately.
While the previous justification provides insights about how Algorithm 1 can identify the direction of causation by bounding forward and backward SDR’s, they do not guaranty identifiability of the above generative model using SIC-based causal inference, which turns out to be non-trivial.
Identifiability can be proved by bounding the gap of the forward-backward inequality (11) as follows.
Lemma 1 Suppose the model from Theorem 2 with fixed input PSD S xx is given, and assume that the filter coefficients b 1 , b 2 , . . .
, b m are independently drawn from some distribution P B with
E [ B ] = 0 and variance E
[ B 2 ] = 1 and finite E [ | B | 3 ] .
Then ρ X → Y ρ Y → X converges to zero in probability for m → ∞ .
This lemma allows to show, for the first time, identifiability with high probability when the number of filter coefficients gets large.
Theorem 3 Given the sampling of coefficients in Lemma 1 and the assumptions there, then the probability for ρ X → Y > ρ Y → X tends to 1 for m → ∞ (as the dimension m of the filter increases), i.e. the direction of causation is identifiable with probability converging to 1 .
6. Robustness to downsampling
In the context of time series, the issue of robustness of causal discovery methods to resampling has been raised in several papers (Gong et al., 2015; Palm and Nijman, 1984; Harvey, 1990).
In particular, Granger causality is known to have issues with downsampling, and specific correction procedures are required (Gong et al., 2015) to infer causal relations.
Here we provide a theoretical result supporting that SIC causal inference is robust to the classical decimation procedure performed in discrete signal processing using an ideal anti-aliasing low pass filter (Crochiere and Rabiner, 1981).
6.1. Decimation procedure
Our decimation setting is provided in Fig.
3, showing that both the cause and the effect time series are low-pass filtered with an ideal anti-aliasing filter and then decimated by a factor D .
The frequency response of the ideal anti-aliasing filter satisfies
[Equation]
and the decimation blocks convert a signal s = { s k } k ∈ Z into ˜ s = ↓ D s =
{ s kD } k ∈ Z (i.e. by picking the value of one time point of every D points).
Using classical decimation results (Crochiere and
Rabiner, 1981), we can derive that
[Equation]
[Figure]
Noticing the absence of overlap (see illustration Fig.
3 between the support of each term of the above periodic summation), we get
[Equation]
We will now investigate whether the true causal direction X
→ Y can be inferred from the decimated observations only.
6.2. Identifiability of the decimated model
Although the decimation certainly involves a loss of information of the measured time series, the following theorem suggests that the SIC can be preserved by such an operation.
Theorem
Assume the forward generative model of the previous section, and ( (cid:101) X , (cid:101) Y ) the time series resulting from the decimation of this model by an integer factor D using an ideal anti-aliasing filter.
Then for any given ϵ with 0 < ϵ < 1 / 2 we have
[Equation]
with probability δ :=
(1 − exp( κ ( m − 1) ϵ 2 )) 2 , where κ is a positive global constant (independent of m , and ϵ ) and K = max |
[Equation]
|
[Equation] [Equation]
D S xx (
[Equation]
(cid:82)
[Equation]
As a consequence of Theorem 4, even after decimating the signal, the concentration of the forward SDR around one is guarantied for high dimensional impulse responses.
Note also that the forward-backward inequality also holds for decimated data, such that the overall identifiability properties are preserved.
However, as intuitively expected, identifiability for an increasing decimation factor D will progressively deteriorate, as the decimation procedure stretches the input and output PSDs from the interval [0 , 1 / 2] to [0 , 1 / (2 D )] .
The estimated frequency response is then also stretched, and its coefficient of variation CV ( | (cid:98) h
˜ X
→ ˜ Y | 2 ) converges to zero as D increases, provided the frequency response respects some smoothness assumptions (e.g. bounded derivative), deteriorating the bound of the forward-backward inequality and thus making the identification of the true causal direction harder.
Qualitatively, indentifiability properties are preserved as long as
S xx has enough variance on the interval [0 , 1 / (2 D )] .
7. Extension of SIC through invariance principles
While a coherent theoretical framework has been presented in the above sections, whether spectral independence is a valid assumption in a given empirical setting remains to be addressed.
Notably, one can question the choice of white noise regular distributions introduced in Section 4.
Indeed, the departure from this reference is exploited to quantify orthogonality, and as such, implicitly reflect an assumption about the considered problem.
To shed light on this assumption, the group invariance perspective on ICM is helpful (Besserve et al., 2018), and we develop it for the case of SIC in Appendix C.
This view justifies the concept of whitening as a way to adapt the SIC framework to datasets where a different set of regular distributions is more appropriate.
To ease readability, we justify this whitening in main text based on the following informal arguments.
The
IGCI perspective of Sec.
4 shows that, by application of corollary 1, causes/inputs that belong to the set of regular distributions trivially satisfy SIC for any choice of mechanism (i.e. filter).
These regular distributions satisfy an invariance property: PSDs are invariant to translations along the frequency axis.
It is thus natural to consider that SIC is suited to applications where signals with such invariance are uninformative.
In contrast, this is not suited for the cases in which such invariance is atypical, as the irregularities quantified by SIC will be blind to this information.
7.1. SIC for power law biological signals
Many biological signals exhibit power law-like PSDs, which decay proportionally to a positive power of the frequency.
This is particularly the case for brain electrical activity (He et al., 2010).
Translation invariant PSDs appear clearly as atypical in such contexts.
However, this can be fixed by preprocessing the recorded signals using an invertible whitening filter w such that its squared frequency response | (cid:98) w ( ν ) | 2 applies an amplifying factor to each frequency that corrects the tendency of the observed signal to have lower power in high frequencies.
This procedure generates a proper group invariance framework as introduced by Besserve et al.
(2018), and corresponds to replacing white noises in Sec.
4.3 by regular distributions with PSDs of the form
[Equation]
which now are representative/typical of the power-law behavior of the considered signals.
SIC can then simply be applied to the preprocessed signal in order to assess ICM in a way that is adapted to electrical brain activity.
7.2. Experiments
To test this approach, we use the neural recording dataset studied in (Shajarisales et al., 2015).
In short, these are recordings of Local Field Potentials (LFP) from two subfields of the rat hippocampus, CA3 and CA1, known to have a clear directed monosynaptic connections CA3 → CA1 that we thus consider as the ground truth causal relationship.
Figure 4(a) depicts the PSD values for recordings from several electrodes in CA3 computed using Welch’s method.
The empirical distribution of
PSD values form this sample do not follow a (frequency)-translation invariant distribution, as the higher frequencies have a much lower power (note the logarithmic scale of the plot).
To correct this property, we apply the above-described whitening transformation, by choosing the inverse of the empirical average PSD over the full dataset (including both recordings form CA3 and CA1.
Frequency (Hz)
PS D electrode 1 electrode 2 electrode(a) 101010 −
PS D
Frequency (Hz) electrode 1 electrode 2 electrode(b)      6   8 x
SDR value ρ CA3 → CA1 ρ CA3 → CA (c) 0.4 0.6 0.
1.2135 6 x
SDR value ρ CA3 → CA1 ρ CA3 → CA (d)
Examples of the resulting whitened PSDs are shown on Fig. 4(b), where we can observe PSD imbalance across frequencies is largely attenuated, suggesting the distribution of whitened PSDs is closer to being frequency translation invariant.
To apply the SIC causal inference approach, we estimated the SDR in both causal (CA3 → CA1) and anti-causal (CA1 → CA3) directions.
While the
SDR distribution of original data in the causal direction is widespread (Fig. 4(c)), it becomes much more concentrated around 1 after whitening (Fig. 4(d)), suggesting that the SIC assumption is “on average valid” after whitening the signals, in line with eq.
(25) in Appendix C. The SDR distribution in the anticausal direction also becomes bounded to values strictly inferior to 1, as predicted by identifiability results (Shajarisales et al., 2015).
Finally, the performance of causal inference using SIC before and after whitening is compared.
The distribution of average performance results across time for all pairs of electrodes is represented by box plots on Supplemental Fig. 8, showing that after whitening the performance is more consistent across electrodes than before: lower tale of the distribution spreads much less towards the low performance values, and average increases significantly from 69.0% to 82.8% after whitening (significant paired signed rank test, p < 10 − 25 ).
8. Discussion
We investigated theoretical foundations of the SIC postulate.
The information geometric view shows that SIC formalizes an independence of input and mechanism via an ‘orthogonality of irregularities’ relative to a set of regular distributions of Gaussian white noises.
We further show that SIC allows identifying the true causal direction in a toy setting, and that this criterion is robust to downsampling.
Finally, we show that the choice of ‘regular distribution’ can be adapted to the application domain.
This set of results clarifies the conditions under which SIC appropriately infers causality based on empirical data.
Extending the framework to systems with noise, non-linearities, hidden confounders and multiple dimensions are key next steps to establish this methodology as a standard tool for time series analysis.
Acknowledgments
This work was partially supported by the German Federal Ministry of Education and Research (BMBF): T¨ubingen AI Center, FKZ: 01IS18039B; and by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645.
Appendix A. Proofs
This section includes proofs of main text results and additional necessary results.
A.1. Consequence of Assumption
Due to basic properties of the Fourier transform, we have the following implications for the model in Assumption 1.
Proposition 3 Assumption 1 implies
• The input and output PSDs S xx and S yy are real 1-periodic bounded continuous functions taking values on [ a, max ν ∈I S xx ( ν )] and [ b,
max ν ∈I S xx ( ν )] respectively, for some a, b > 0 .
• The forward and backward frequency responses (cid:98)
h
X
→ Y and (cid:98) h Y
→ X are complex bounded 1periodic continuous functions such that | (cid:98)
h X
→ Y | 2 = S
yy S
xx
>
c and | (cid:98)
h X
→ Y | 2 = S xx S yy >
d for some c, d > 0 .
Proof For any sequence a ∈ ℓ 1 ( Z ) there is uniform convergence and boundedness of the Fourier transform series.
As a consequence, all Fourier transforms considered well defined, periodic, continuous and bounded.
S xx being strictly positive, by continuity on one period (as Fourier transform of the autocorrelation function), its minimum is strictly positive.
BIBO stability of h X
→ Y entails that S yy is also well defined, periodic, continuous and bounded (e.g. using the above Proposition 1).
Then BIBO stability of h Y → X entails that its Fourier transform S xx /S
yy is bounded, which implies by contradiction that inf v S
yy ( v ) must be strictly positive.
Finally, strictly positive bounds on S xx and S
yy imply strictly positive upper and lower bounds for the frequency responses.
A.2. Proof of Theorem
First, we start by finding the divergence rate for a given Gaussian process from the set of white
Gaussian noises (i.e. the Gaussian processes with constant spectra).
Lemma 2 Let ¯ E to be the set of discrete white Gaussian noises.
Assuming X is a Gaussian process with C xx ∈ ℓ 1 ( Z ) and such that S xx is strictly positive at all frequencies, then the projection P ¯ E X of X on ¯ E is a Gaussian white noise of power P ( X ) , and the corresponding divergence is
[Equation]
The formula also holds when exchanging X for Y
To get a intuition of (12) note that q ( ν ) := S xx ( ν ) P ( X ) can be formally interpreted as probability density on [ − 1 / 2 , 1 / 2] due to (cid:82) 1 / 2 − 1 / 2 q ( ν ) dν = 1 .
If u with u ( ν ) = 1 denotes the constant density, (12) can be rephrased as the relative entropy distance (cid:82) 1 / 2 − 1 / 2 u ( ν ) log( qν ) u ( ν )
=
D ( u ∥ q ) .
In other words, it measures to what extent q deviates from the uniform distribution.
Based on this Lemma, we follow the steps of (Janzing et al., 2012) to prove the theorem.
Proof We denote elements of the set of white Gaussian noise distributions
U α , where α denotes the power of U α and hence the constant value of its corresponding PSD.
The Gaussian noise distribution with minimum distance corresponds to the value of α for which the derivative of ¯ D ( P X ∥
U α ) vanishes.
We can compute the distances via Proposition 1 because condition ( ii ) is satisfied:
[Equation]
Hence, the derivative vanishes for (cid:82) 12 − 12 S xx ( ν ) α= 1 α , which amounts to α = P ( X ) .
We thus get
[Equation]
Using the definition of P we obtain:
[Equation]
which proves our claim.
The formula also holds for Y because a linear filter maps a Gaussian process on a Gaussian process.
Proof of Theorem 1:
Using Lemma 2 we have
[Equation]
dν and ¯ D
[Equation]
Transforming U X and h t to Fourier domain it is easy to see that → P Y is the distribution of a stationary
Gaussian process with PSD P ( X ) | ˆ h ( ν ) | 2 according to eq.
Therefore using Proposition 1 we get:
[Equation]
which completes the proof.
□
A.3. Proof of Corrolary
Proof Assume SIC is satisfied, then the last term of eq.
(9) vanishes, yielding
[Equation]
Moreover, SIC also implies that the projection of → P Y on ¯ E is U Y .
This is because SIC implies that →
P Y and P Y have the same power.
Hence, we get
[Equation]
For the converse implication, assume we have orthogonality of irregularities, combining eq.
(10) with eq.
(9) we get
[Equation]
Using Lemma 2 to get an analytic expression for the left-hand side of the above equation, and eq.
(8) for the right-hand side this yields −
[Equation]
[Equation]
which after simplification, and noting that ⟨| ˆ h | 2 ⟩ = ⟨ S yy S xx ⟩ , yields
[Equation]
which is equivalent to SIC.
A.4. Proof of Lemma
Proof We denote ˆ h mb ( ν ) the transfer function (cid:98)
h X
→ Y for the model such that
[Equation]
Let B 1 , B 2 , ... be a sequence of i.i.d.
real-valued variables on a probability space (Ω , Σ , P Ω ) .
Then we first define for each frequency ν the sequence (cid:16) (cid:98) B mν (cid:17) m ∈ N of random variables via
[Equation]
Using know vector valued central limit theorems, (cid:98) B mν converges to Gaussian on the complex plane for each ν .
Now define for each ω ∈ Ω the sequence (cid:16) (cid:98) B m ( ω ) (cid:17) m ∈ N of random variables on the probability space ( I , B , λ ) with B denoting the Borel sigma algebra and λ the Lebesgue measure (to formalize the random choice of frequency), via
[Equation]
Theorem 1 in (Janzing et al., 2017) states that this sequence of complex random variables converges for m → ∞ to an isotropic two-dimensional Gaussian Z on the complex plane where real and imaginary parts are independent with variance 1 / 2 .
More precisely, the random variable
[Equation]
converges in probability to zero for any pseudo-metric d that is well-behaved in the sense of Definition 1 in (Janzing et al., 2017).
We now define a pseudo-metric on the set of distributions on C by
[Equation]
This pseudo-metric is well-defined because
[Equation]
is well-behaved for the set of distributions on R , see remarks after Definition 1 in (Janzing et al., 2017).
We now show that
[Equation]
in probability.
First, note that
[Equation]
due to the strong law of large numbers.
Recall that | Z | 2 is χ 2 -distributed with one degree of freedom, which implies that its density has infinite slope at 0 .
Hence, for any arbitrarily large c we can always find a sufficiently small ϵ such that
[Equation]
Form Markov’s inequality for a > 0 we get
[Equation]
such that
[Equation]
For sufficiently large m we thus can ensure that
[Equation]
with probability at least 1 − δ for any arbitrarily small δ .
This is because, as stated above, Theoremin (Janzing et al., 2017) entails that P (cid:98) B m ( ω ) converges to the distribution of P Z with respect to any well-behaved pseudo-metric, in particular with respect to d defined in (13).
Combining (17) and (16) shows that
[Equation]
with probability at least 1 − δ .
Using also (14) we conclude that
[Equation]
is larger than c/ 2 also with probability 1 − δ for sufficiently large m .
As c was chosen arbitrary large, this proves that
[Equation] Hence the inverse of the above quantity converges to zero in probability, which shows that [Equation]
A.5. Proof of Lemma
The following lemmas will be helpful in proving Theorem 2.
Lemma 3 (Serre, 2010)
For a given Hermitian matrix H and any principal submatrix of H , H ′ , their spectral radius ρ s satisfies
[Equation]
Lemma 4 (Gray, 2006) Let f : [ − 12 , 12 ) → R f ∈
L 1 be a bounded function and suppose t k is its
Fourier series coefficients, i.e. t k = (cid:90) [Equation] f
[Equation]
πkν dν [Equation] t ∈ Z
[Equation]
Consider Toeplitz matrices T n defined as [ T n ] ij = t i − j i, j ∈ { 0 , ..., n − 1 } with eigenvalues τ n,k (0 ≤ k ≤ n − 1) .
Then if t i are absolutely summable we get: min x
[Equation]
In addition, we exploit a concentration of measure result for Lipschitz continuous functions of random rotation matrices:
Lemma 5 (Guionnet, 2009, Corollary 6.14)
For any differentiable function f : SO ( N ) → R such that, for any X,
Y ∈ SO
( N ) , | f
( X ) − f ( Y ) | ≤ K ∥ X − Y ∥ F , where ∥∥ 2 denotes the Forbenius norm, we have for any random matrix O Haar distributed on SO ( N ) and for all δ ≥
[Equation]
with probability at least 1 − 2 e − Nδ
K thus represents a Lipschitz constant of f .
This concentration result implies another one more specific to our case.
Lemma 6 Let A, B be a square matrices in R n × n , assuming A is symmetric and let O be a random rotation matrix Haar distributed on SO ( n ) .
For all δ ≥
[Equation]
with probability at least 1 − 2 e − n 3 δ 2 , and
[Equation]
with probability at least 1 − 2 e − n 2 δ
Proof The proof follows two steps in order to apply the above lemma to the function O
(cid:55)→ τ n ( AOBO T ) restricted to SO ( n ) .
Step 1: First, we show O (cid:55)→ τ n ( AOBO T ) is Lipschitz continuous on SO ( n ) .
Let U, V ∈
SO ( n ) , then
[Equation]
applying the Cauchy-Schwartz inequality on the two Frobenius scalar products ( | Tr ( XY T ) | ≤ ∥ X ∥ F ∥ Y ∥ F ) we get
[Equation]
Using the matrix norm inequality 3 ∥ XY ∥ F ≤ ∥ X ∥ F ρ ( Y ) we get
[Equation]
and finally by rotation invariance of the spectral norm
[Equation]
Step 2: Now we show E O τ n ( AOBO T ) =
τ n ( A ) τ n ( B ) .
By linearity of the trace we have
[Equation]
It is easy to see that E
[ OBO T ] commutes with every Q ∈ SO ( n ) due to Q E
[ OBO T ] Q T =
E
[ OBO T ] .
Thus, E
[ OBO T ] is a multiple of the identity (due to Schur’s lemma, otherwise
SO ( n ) would not act irreducibly on R n ).
Finally, we conclude E
[ OBO T ] =
τ n ( B ) 1 because τ n ( E
[ OBO T ]) =
τ n ( B ) .
Finally, combining the results from both steps we can apply Lemma 5, we get, using the above
Lipschitz constant K = 2 n ∥ A ∥ F ρ ( B )
[Equation]
with probability at least 1 − 2 e − nδ 2 , which gives the first bound of the lemma.
The second bound is obtained using the matrix norm inequality ∥
A ∥ F ≤ √ nρ
( A ) .
Now we are ready to prove Theorem 2:
Theorem 2 (Concentration of measure for FIR filters) Assume a length m mechanism’s impulse response
h X
→ Y with a spherically symmetric generative model and a fixed input PSD S xx , then for any given ε > 0 , with probability at least 1 − 2 exp( − m 3 ε 2 ) , we have
[Equation]
Proof Without loss of generality and for the sake of simplicity we only consider the positive indices of the time series and we take the filter to be causal; other cases can be treated in a similar way.
Then the following relation holds between input and output of the filter:
[Equation]
Formulated in terms of matrices the above relation can be represented as
[Equation]
where H N,m is a N × ( N + m − 1) matrix as follows    b m −b m −· · · b · · ·   b m −· · · bb· · ·  . . .
· · · b m −· · · bb   · · · b m −· · · bb  
We define Σ iX ∈
M m × m ( R ) to be the covariance matrices as follows:
[Equation]
Since the time series under consideration are weakly stationary that Σ iX is independent of i and we denote it Σ m .
If we take Σ X 0: N − 1 , Σ Y 0: N − 1 ∈ M N × N ( R ) to be the covariance matrices for X 0: N −and Y 0:
N − 1 respectively, then we have
[Equation]
Also define Σ UY 0: N − 1 to be the covariance matrix of the output for FIR S ′ with b ′ =
U ⊤ b .
Furthermore assume the spectrum of the output for this filter is S Uyy .
One can write the diagonal elements of Σ UY 0: N − 1 based on the above equation as follows:
[Equation]
which is therefore equal to the normalized trace of Σ Y
[Equation]
N − [Equation]
Due to the spherical invariance assumption, b is a single sample from the product R U
(Bryc, 2012, Theorem 4.1.2).
Moreover, U can obviously be rewritten as the first column of a random rotation matrix, U
=
O x 0 , where x 0 is the canonical basis vector with first component 1, while other components are zero, and O is a random matrix Haar distributed on SO(m).
Replacing b by its corresponding random variable, we get
[Equation]
Taking A = x 0 x 0 ⊤ and B = Σ m in Lemma 6 we get
[Equation]
with probability at least 1 − 2 e − m 3 δ 2 .
Moreover, we notice that according to the generative model, the squared norm of the impulse response is sampled from the random variable R 2 , hence
[Equation]
with the same probability as above.
On the other hand the elements of diagonals of Σ m are C X
Therefore:
[Equation]
Since Σ is a principal submatrix of Σ X 0: N − 1 , by Lemma
[Equation]
Because C X ( τ ) ’s are absolutely summable, based on Lemma 4 we get
[Equation]
and then inequality (20) can be rewritten as
[Equation]
which completes the proof.
A.6. Proof for Proposition
First, we derive to helpful lemmas that are used to prove the proposition; the first one being used to prove the first part and the second lemma is used to infer the second part of the proposition above.
Lemma 7 For f ∈ L 2 ( I ) non-constant, such that 1 /f ∈ L 2 ( I ) , we have
[Equation]
Proof The inequality follows from Jensen’s inequality, stating
E [1 /W ] < 1 /E
[ W ] , for any non-constant random variable W .
Lemma 8 Let f ∈ L 1 ( I ) be positive, non-constant
, such that 1 /f ∈ L 1 ( I ) and (cid:82) I f ( x ) dx = 1 .
Assume ∃ α > 0 , ∀ x ∈ I , f ( x ) ≤ 2 − α , then (cid:90) I f ( x ) [Equation] · (cid:90)
[Equation]
f
[Equation]
x ) −
[Equation]
dx
Proof We denote s ( x ) =
f
( x ) − 1 .
Then (cid:82) I s ( x ) dx = [-
-]0 and (cid:90) I f ( x ) dx.
[Equation]
For x > − 1 , we have
[Equation]
Replacing s ( x ) with x in (21) we get:
[Equation]
Since 1 − s ( x ) = 2 − f ( x ) ≥ α > 0 ,
[Equation]
Now we can prove Proposition 2.
Proposition 2 (Forward-backward inequality) For a given linear filter S with impulse response
h X
→ Y , input PSD S xx and output PSD S
yy , let S have a non-constant modulus frequency response (cid:98)
h X
→ Y , and assume there exists α > 0 such that for all ν ∈
I ,
[Equation]
Then
[Equation]
Proof (i) Using the definition of Spectral Dependency Ratios and Lemma 7 it easily follows that
[Equation]
(ii) Applying Lemma 8 to
[Equation]
we get inequality (11).
A.7. Proof of Theorem
Proof Lemma 1 implies for any fixed ϵ >
[Equation]
as the dimension of the filter increases.
Moreover, Theorem 2 implies
[Equation]
Given that
P ( | ρ Y → X | < 1 / 2) ≥ P ( { ρ Y → X .ρ X → Y < 1 / 4 } & { ρ X → Y > 1 / 2 } ) = 1 − P ( ρ Y → X .ρ X → Y ≥ 1 / 4) − P ( ρ X → Y ≤ 1 / 2) ,
P ( | ρ Y → X | < 1 / 2) thus tends to one as m → + ∞ .
Moreover
P ( ρ Y → X < ρ X → Y ) ≥ P ( {| ρ X
→ Y | > 1 / 2) } & {| ρ Y → X | < 1 / 2 } )
= 1 − P ( {| ρ X
→
Y | ≤ 1 / 2) } or {| ρ Y → X | ≥ 1 / 2 } ) ≥ 1 − P ( | ρ X → Y | ≤ 1 / 2) − P ( | ρ Y → X | ≥ 1 / 2) , such that P ( ρ Y → X < ρ X → Y ) tends to one
a m tends to infinity.
A.8. Proof of Theorem
For convenience we restate the theorem here:
Theorem 4 Assume the forward generative model of the previous section, and ( (cid:101) X , (cid:101) Y )
the time series resulting from the decimation of this model by an integer factor D using an ideal anti-aliasing filter.
Then for any given ϵ with 0 < ϵ < 1 / 2 we have
[Equation]
with probability δ :=
(1 − exp( κ ( m − 1) ϵ 2 )) 2 , where κ is a positive global constant (independent of m , and ϵ ) and K = max | ν | < 1 /[-
-]
[Equation]
xx
[Equation]
Let us first write the forward SDR of the downsampled data:
[Equation]
Second, we can apply the previous concentration of measure result of (Shajarisales et al., 2015)
(Theorem 1) to the low pass filtered causes and effects as they satisfy the requirements of the generative model, then the corresponding forward SDR is
[Equation]
and we have
[Equation]
with probability at least 1 − exp( − κ ( m − 1) ε 2 ) .
Moreover, we have (using h =
Ub and U H U
= I m ):
[Equation]
As a consequence:
[Equation]
| ρ a ∗ X → a ∗ Y
[Equation]
a ∗ X → a ∗
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Hence the bound:
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Only the rightmost term remains to be bounded.
We will thus proceed to the evaluation of (cid:82) 1 / {+
+}
D| (cid:98) h ( ν ) | 2 dν ; first by estimating the integral using a left point approximation using a grid of size M (we choose M as a multiple of 2 D ):
[Equation]
By using a bound on the derivate of (cid:98) h ( | (cid:98) h ′ | ≤ √ m 3 2 π ∥ b ∥ )
we can bound the left point approximation:
[Equation]
Now L DM ( (cid:98) h ) can be evaluated using the concentration of measure theorem from (Janzing et al.
Let F M,m be the matrix implementing the M frequency points discrete Fourier Transform (DFT) of an m-time points vector 4 such that F M,m = { exp − 2 i πkn/M } k =0 ..
M − 1; n =0 ..m − 1 , and let L M,D the diagonal matrix such that ( L M,D ) kk = 1 k<M/ (2 D ) (), we have
[Equation]
As a consequence, we get the following concentration of measure result, with probability δ = 1 − exp( − κ ( m − 1) ϵ 2 )
[Equation]
where ∥·∥
o denotes the operator norm [Equation] which can be bounded as follows:
[Equation]
Moreover ∥ L M,D ∥ o = 1 and the operator norm of the zero padded FFT matrix can be bounded by writing it down using the full M-dimentional FFT matrix, such that F M,m = F M,M
[Equation]
As a consequence we get the bound (noticing Tr ( F HM,m L M,D F M,m ) =
mM/ (2 D ) ):
[Equation]
such that
[Equation]
We note that this bound does not depend on M , such that by taking the limit M → ∞ we can bound (cid:82) 1 / 2 D
[Equation]
Putting together all bounds we get
[Equation]
and using again the previous bound we finally get:
[Equation]
with probability δ 2 = (1 − exp( − κ ( m − 1) ϵ 2 )) 2 and for ϵ < 1 /.
Appendix B. Additional background B.1. Fourier transform of sequences
Consider a sequence of real or complex numbers a = { a t , t ∈ Z } as a deterministic vector from the ℓ 1 ( Z ) sequence space of bounded ℓ 1 norm,
∥ a ∥ 1 = (cid:80) t ∈ Z | a t | < ∞ .
The support of the sequence is the subset Supp ( a ) = { k ∈ Z , | a k | > 0 } ⊂ Z .
Its Discrete-Time Fourier Transform (DTFT) is defined as
[Equation]
Note that the DTFT of such sequence is a continuous 1-periodic function of the normalized frequency ν and can be characterized by its values on any unit length interval.
We will use the unit interval centered around zero [ − 1 / 2 , 1 / 2]
=: I .
In case the original sequence is real-valued, its
Fourier transform is conjugate symmetric ( (cid:98) a ( − ν )
= (cid:98) a ∗ ( ν ) )
such that its modulus is an even function.
This will be the case in the remainder of this paper.
The squared ℓ 2 -norm ∥ a ∥ 22 = (cid:80) t | a t | 2 is often called energy .
By Parseval’s theorem, it can be expressed in the Fourier domain by ∥ a ∥ {+2 
+} = (cid:82) 1 / 2 − 1 / 2 | (cid:98) a ( ν ) | 2 dν .
To simplify notations, we will denote by ⟨ .
⟩ the integral (or average) of a function over the unit interval I , such that ∥ a ∥ 22 = ⟨| (cid:98) a | 2 ⟩ .
B.2. Probabilistic interpretation
To motivate why eq.
(3) is called an independence condition, we notice that the difference between the left and the right hand side can be written as a covariance:
[Equation]
To understand the rephrasing as covariance , note that the maps ν (cid:55)→ S xx ( ν ) and ν (cid:55)→ | (cid:98)
h X
→ Y ( ν ) |can be formally interpreted as random variables on the probability space [ − 1 / 2 , 1 / 2] equipped − 0. − 0. − 0.
− 0. with the uniform measure. From the purely mathematical point of view, this interpretation is certainly justified because any measurable map from a measure space to R can be considered as a random variable.
For a more intuitive approach, one can think of a random experiment where one chooses a frequency ν ∈ [ − 1 / 2 , 1 / 2] according to the uniform distribution and then observes
S xx ( ν ) and | (cid:98) h X
→ Y ( ν ) | 2 .
Finally, after a large (actually infinite) number of runs, the covariance of these quantities is given by (22).
Accordingly, SIC can be interpreted as follows:
The expectation of S xx ( ν ) | (cid:100) h
( ν ) X
→ Y | 2 is the same as if two frequencies ν
and ν 2 were drawn independently (as illustrated on Fig. 5) for both functions, which amounts to match at random the frequencies of the amplification factor and of the input PSD.
Conversely, SIC would be violated, for instance, if the system mainly amplifies the frequencies with low intensity, as it happens for the ‘anti-causal’ direction in our motivating example in Subsection B.3.
However, spectral independence does not correspond to classical statistical independence for two reasons.
First, “independence” is measured at the level of the parameters of the causal system (filter coefficients and input PSD properties) and not at the level of the observed random variables.
Second, statistical independence is a stronger statement than uncorrelatedness, and SIC corresponds essentially to the latter.
B.3. Intuitive Example
As illustrated in Fig. 6, assume that X is a white noise process, such that it has a spectral density function with uniform distribution, i.e. S xx ( ν )
=
c > 0 for all ν ∈ [ − 1 / 2 , 1 / 2] .
Here, we reject the hypothesis that Y causes X because it would require the corresponding filter mapping Y to X to be tuned relative to the input signal Y : it amplifies the frequencies having small power in the input signal while it weakens those that have large power ‘in order’ to generate an output whose power is uniformly distributed over the frequencies.
This results in | (cid:98)
h Y
→ X | 2 and S yy being strongly anticorrelated (see Fig.
6 bottom right panel).
Thus, we have an extreme example of violation
[Figure]
[Equation]
⌘ .
[Equation]
⌘ .
Y(t)
S( X +
X 0 )(t) =
S(X )(t) +
S(X 0 )
(t)
S(X (t; )) = S(X ) (t; ) h(t) Frequency
Frequency bh Y ) X | [Equation] ⌘ . [Equation]
|bh
[Equation]
Y |
[Equation]
⌘ .
Y(t)
S( X +
X 0 )(t) =
S(X )(t) +
S(X (t; )) =
S(X )
(t; h(t) Frequency
[Figure]
of ICM: S
yy encodes all the second order statistics of Y , the hypothetical input (for zero mean
Gaussian time series it even describes the statistics of the input signal completely).
On the other hand, (cid:98) h Y
→ X is a representative feature of the mechanism.
Thus, the mechanism that generates X from Y is informative about its input.
Appendix C. SIC and group invariance Group invariance perspective on ICM
The group invariance framework assesses ICM by quantifying the genericity of the relationship between input x , corresponding to the PSD of the putative cause time series in the context of this paper, and mechanism m , corresponding to the filter.
As explained in (Besserve et al., 2018), this requires defining two objects: (1) the generic group G is a compact topological group that acts on cause properties (i.e. elements of the group are transformation that modify the properties of the cause), thus equipped with a unique Haar probability measure µ G , (2) the contrast C is a real valued function.
The contrast and generic group introduced in such a way allow to compute the expected contrast value when randomly “breaking” the cause-mechanism relationship using generic transformations according to the following definition.
Defnition 1 Given a contrast C , the Expected Generic Contrast (EGC) of a cause mechanism pair ( x, m ) is defined as:
[Equation]
The relation between m and x is G -generic under C , whenever
[Equation]
24) holds approximately.
Equation (24) is the genericity equation , which expresses the idealized ICM postulate (hence the term “holds approximately”).
Besserve et al.
(2018) give a probabilistic interpretation of the concept of genericity.
Assume we are given a generative model such that the cause x is a single sample drawn from a meta-distribution 5 P X (see Fig. 7).
To estimate genericity irrespective of the possible values of x , we consider the genericity ratio C ( mx ) / ⟨ C ⟩ m,x : this quantity should be close to one with high probability in order to satisfy ICM assumptions.
Assume P X is a G -invariant distribution, under mild assumptions (Wijsman, 1967) x can be parametrized as x = g ˜ x where g is a sample from a µ G -distributed variable G , and ˜ x is a sample from anther RV independent of G .
[Equation]
This tells us that the postulate of genericity is valid at least “on average” for the generative model.
On the contrary, if this average would be different from 1 as it may happen for a non-invariant
P X , the postulate is unlikely be valid for individual examples.
As represented on Fig.
7, the same reasoning can be applied when sampling the mechanism from an invariant distribution.
The case of SIC.
By using the power of the time series Y in (1) as a contrast, and frequency translations as the generic group, we can show that Spectral Independence postulate correponds to a genericity equation in the group invariance framework.
Proposition 4 Let G be the group of modulo 1/2 translations that acts on the PSD by shifting its graph for positive frequencies ( ν ∈ [0 , 1 / 2] ) while the graph for negative frequencies is defined so that the transformed PSD is even.
Using the total power as a contrast, G− genericity is equivalent to SIC.
Proof Suppose that for a given mechanism m and given input S xx the G− genericity assumption is satisfied.
Noticing that µ G is the uniform probability measure over [0 , 1 / 2] .
This amounts to (cid:90) 1 /− 1 /
[Equation]
ν ) | 2 dν = (cid:90) 1 / (cid:32) (cid:90) 1 / | (cid:98) h
( ν ) | 2 S xx ( ν − g ) µ G ( g ) dν (cid:33) dg =(cid:90)
1 / (cid:90) 1 / | (cid:98)
h
( ν ) | 2 S xx ( ν − g ) dνdg =(cid:90) 1 / | (cid:98) h
( ν ) |(cid:32)(cid:90) 1 /
S xx ( ν − g ) dg (cid:33) dν
= (cid:90) 1 /− 1 /
S xx ( ν ) dν · (cid:90) 1 /−
/| (cid:98) h(
[Equation] ) | 2 dν
This corresponds to the formula of the SIC postulate.
□
Whitening: a way to enforce group invariance.
As can be understood from eq.
(25) one possible source of misfit between ICM based approaches and real data is a lack of G -invariance of the actual generating mechanism behind the observed data.
In this section, we suggest a simple approach to adapt ICM methods to “baseline” properties of the set of causes.
The genericity equation is compatible with a G -invariant probabilistic generative model of the cause.
Such an invariance assumption can be checked on real data, for example by verifying the G -invariance of the empirical average of all putative causes.
We can then seek to correct any lack of invariance: assume
b the average of [Figure] y empirically observed causes is not group invariant.
If there exists an invertible transformation w such that w ◦ b = u becomes invariant, we can define a new group invariance framework with
[Equation]
as new generic group and C w : x (cid:55)→ C ( w ◦ x ) as a new contrast.
w is called a whitening transformation as it maps a predefined average distribution into a “white” invariant average distribution.
Appendix D
Supplemental
[Figure]