Causal Disentanglement for Time Series
Author s
Abstract
Recently, in the field of nonlinear Independent Component Analysis (ICA), strongidentifiability results for disentanglement have been established by using certainside information, such as class labels or history information for time series, in addition to independence.
However, most existing work is constrained by functionalform assumptions such as stationary independent sources or further with lineartransitions and distribution assumptions such as exponential family distribution.
It is unknown whether the underlying latent processes and their causal relationsare identifiable if they have arbitrary, nonparametric causal influences in between.
We propose a principled framework called LCD-NM to recover time-delayed latent causal variables and identify their relations from measured temporal dataunder stationary environments and under different distribution shifts.
Specifically,the framework factorizes unknown distribution shifts into transition distributionchanges caused by fixed dynamics, time-varying latent causal relations, and globalchanges in observation.
We establish the identifiability theories of nonparametriclatent causal processes from their nonlinear mixtures under fixed dynamics andanalyze how distribution changes can further benefit the identifiability.
Throughexperiments, we show that time-delayed latent causal influences are reliably identified and that our approach considerably outperforms existing baselines that do notcorrectly exploit this modular representation of changes.
Our results demonstratethat disentanglement in time-series settings seems promising both in stationary environments and general nonstationary environments, in which the latent processeshave nonparametric causal influences in between.
Introduction
Causal discovery from time-series data is a crucial task in numerous fields [1, 2, 3].
Most existingwork focuses on estimating the temporal causal relations among observed variables.
However, inmany real-world scenarios, the observed signals (e g., image pixels in videos) do not have directcausal edges but are generated by latent temporal processes or confounders that are causally related.
Inspired by these scenarios, this work aims to uncover causally-related latent processes and theirrelations from observed temporal variables.
Estimating latent causal structure from observations,which we assume are unknown (but invertible) nonlinear mixtures of the latent processes, is verychallenging.
It has been found in [4, 5] that without exploiting an appropriate class of assumptionsin estimation, the latent variables are not identifiable or disentangled in the most general case.
As aresult, one cannot make causal claims on the recovered relations in the latent space.
Recently, in the field of nonlinear Independent Component Analysis (ICA), strong identifiabilityresults of the latent variables have been established [6, 7, 8, 9, 10] by using certain side information,such as class labels, in addition to independence.
For time-series data, history information is widelyused as the side information for the disentanglement of latent processes.
To establish identifiability,the existing approaches enforce different sets of functional and distributional form assumptions as
[Table] [Table]
constraints in estimation; for example, (1) PCL [7], GCL [8], HM-NLICA [11] and SlowVAE
[12]assume mutually-independent sources in the data generating process.
However, this assumptionmay severely distort the identifiability if the latent variables have time-delayed causal relations inbetween (i.e., causally-related process); (2) SlowVAE [12] and SNICA [13] assume linear relations,which may distort the identifiability results if the underlying transitions are nonlinear, and (3)
SlowVAE [12] assumes that the process noise is drawn from Laplacian distribution; i-VAE [9]assumes that the conditional transition distribution is part of the exponential family.
However, inreal-world scenarios, one cannot choose a proper set of functional and distributional form assumptionswithout knowing in advance the parametric forms of the latent temporal processes.
Our first stepis hence to understand under what conditions the latent causally processes are identifiable if theyhave nonparametric transitions in between.
With the proposed condition, our approach allowsrecovery of latent temporal causally-related processes in stationary environments without knowingtheir parametric forms in advance.
On the other hand, nonstationarity has greatly improved theidentifiability results for learning the latent causal structure [14, 15, 16].
For instance, LEAP [14]established the identifiability of latent temporal processes, but in limited nonstationary cases, underthe condition that the distribution of the noise terms of the latent processes varies across segments.
Our second step is to analyze how distribution shifts benefit our stationary condition and to extendour condition to a general nonstationary case.
Accordingly, our approach enables the recovery oflatent temporal causal processes in a general nonstationary environment with time-varying relationssuch as changes in the influencing strength or switching some edges off [17] over time or domains.
[Figure]
Given the identifiability results,we propose a learning framework, called LCD-NM , to recovernonparametric time-delayed latent causal variables and identify their relations from measuredtemporal data under stationaryenvironments and under nonstationary environments in which itis unknown in advance how thejoint distribution changes acrossdomains (we define it as “unknown distribution shifts”).
Forinstance, Fig.
1 shows an example of multiple video domains ofa physical system under differentmass, gravity, and environmentrendering settings 1 .
With LCDNM , the differences across segments are characterized by thelearned change factors ˆ θ dyn r of domain r (note that domain indexis given to the model) that encode changes in transition dynamics, and changes in observation orstyles modeled by ˆ θ obs r
(we use “causal dynamics” and “latent causal relations/influences” inter- changeably).
We then present a generalized time-series data generative model that takes these changefactors as arguments for modeling the distribution changes.
Specifically, we factorize unknowndistribution shifts into transition distribution changes in stationary processes, time-varying latentcausal relations, and global changes in observation by constructing partitioned latent subspaces, andpropose provable conditions under which nonparametric latent causal processes can be identifiedfrom their nonlinear invertible mixtures.
We demonstrate through a number of real-world datasets,including video and motion capture data that time-delayed latent causal influences are reliably identified from observed variables under stationary environments and unknown distribution shifts.
Throughexperiments, we show that our approach considerably outperforms existing baselines that do notcorrectly leverage this factorized representation of changes.
Our results show that disentanglement intime-series settings seems promising both in stationary and general nonstationary environments, inwhich the latent processes have arbitrary, nonparametric causal influences in between.
Related Work
Causal Discovery from Time Series Inferring the causal structure from time-series data is criticalto many fields including machine learning [1], econometrics [2], and neuroscience [3].
Most existingwork focuses on estimating the temporal causal relations between observed variables.
For thistask, constraint-based methods [18] apply the conditional independence tests to recover the causalstructures, while score-based methods [19, 20] define score functions to guide a search process.
Furthermore, [21, 22] propose to fuse both conditional independence tests and score-based methods.
The Granger causality [23] and its nonlinear variations [24, 25] are also widely used.
Nonlinear ICA for Time Series Temporal structure and nonstationarities were recently used toachieve identifiability in nonlinear ICA.
Time-contrastive learning (TCL [6]) used the independentsources assumption and leveraged sufficient variability in variance terms of different data segments.
Permutation-based contrastive (PCL [7]) proposed a learning framework which discriminates between true independent sources and permuted ones, and identifiable under the uniformly dependentassumption.
HM-NLICA [11] combined nonlinear ICA with a Hidden Markov Model (HMM) to automatically model nonstationarity without manual data segmentation.
i-VAE [9] introduced VAEs toapproximate the true joint distribution over observed and auxiliary nonstationary regimes.
Their workassumes that the conditional distribution is within exponential families to achieve the identifiabilityof the latent space.
The most recent literature on nonlinear ICA for time-series includes LEAP [14]and (i-)CITRIS [26, 27].
LEAP proposed a nonparametric condition leveraging the nonstationarynoise terms.
However, all latent processes are changed across contexts and the distribution changesneed to be modeled by nonstationary noise and it does not exploit the stationary nonparametriccomponents for identifiability.
Alternatively, CITRIS proposed to use intervention target informationfor identification of scalar and multidimensional latent causal factors.
This approach does not sufferfrom functional or distributional form constraints, but needs access to active intervention.
Problem Formulation
3.1 Time Series Generative Model
As a fundamental case , we first present a regular, stationary time-series generative process wherethe observations x t comes from a nonlinear (but invertible) mixing function g that maps the timedelayed causally-related latent variables z t to x t .
The latent variables or processes z t have stationary,nonparametric time-delayed causal relations.
Let τ be the time lag:
[Equation]
Stationary
[Equation]
Note that with nonparametric causal transitions, the noise term ϵ it ∼ p
ϵ i (where
p ϵ i denotes thedistribution of ϵ it ) and the time-delayed parents Pa ( z it ) of
z
it (i.e., the set of latent factors thatdirectly cause z it ) are interacted and transformed in an arbitrarily nonlinear way to generate z it .
Understationarity assumptions, the mixing function g , the transition functions f i and the noise distributionsp ϵ
i are invariant.
Finally, we assume that the noise terms are mutually-independent (i.e., spatially andtemporally independent), which implies that instantaneous causal influence between latent causalprocesses is not allowed by the formulation.
The stationary time-series model in the fundamentalcase is used to establish the identifiability results under fixed causal dynamics in Section 4.1.
We further consider two violations of the stationarity assumptions in the fundamental case, which leadto two nonstationary time series models.
Let u denote the domain or regime index.
Suppose thereexist m regimes of data, i.e., u r with r = 1 , 2 , ..., m , with unknown distribution shifts.
In practice, thechanging parameters of the joint distribution across domains often lie in a low-dimensional manifold[28].
Moreover, if the distribution is causally factorized, the distributions are often changed in aminimal and sparse way [29].
Based on these assumptions, we introduce the low-dimensional minimalchange factor ( θ dyn r , θ obs r ) , which was proposed in [30], to respectively capture distribution shiftsin transition functions and observation.
The vector θ r =
( θ dyn r , θ obs r ) has a constant value in eachdomain but varies across domains.
The formulation of the nonstationary time-series model is in linewith [30].
The nonstationary model is used to establish the identifiability results under nonstationarycases in Section 4.2, where we show that the violation of stationarity in both ways can even furtherimprove the identifiability results.
We first present the two nonstationary cases.
(1) Changing Causal
Dynamics .
The causal influences between the latent temporal processes are changed across domainsin this setting.
We model it by adding the transition change factors θ dyn r as inputs to the transitionfunction: z it =
f i (cid:16) { z j,t − τ | z j,t − τ ∈
Pa ( z it ) } , θ dyn r , ϵ it (cid:17) .
(2) Observation Changes.
The globalproperties of the time series (e.g., video styles) are changed across domains in this setting.
Ourmodel captures them using latent variables that represent global styles; these latent variables aregenerated by a bijection f i that transforms the noise terms
ϵ i,t into the latent with change factor θ obs r :z i,t = f i (cid:16) θ obs r , ϵ i,t (cid:17) .
Finally, we can deal with a more general nonstationary case by combining thethree types of latent processes in the latent space in a modular way.
Modular Distribution Shifts.
[Equation]
The latent space has three blocks z t =( z fix t , z chg t , z obs t ) where z fix s,t denotes thes th component of the fixed dynamicsparts, z chg c,t denotes the c th componentof the changing dynamics parts, andz obs o,t is the o th component of the globalchanges.
The functions [ f s , f c , f o ]capture fixed and changing transitions and observation changes for each dimension of z t in Eq.
3.2 Identifiability of Latent Causal Processes and Time-Delayed Latent Causal Relations
We define the identifiability of time-delayed latent causal processes in the representation functionspace in Definition 1 .
Furthermore, if the estimated latent processes can be identified at least upto permutation and component-wise invertible nonlinearities, the latent causal relations are alsoimmediately identifiable because conditional independence relations fully characterize time-delayedcausal relations in a time-delayed causally sufficient system, in which there are no latent causalconfounders in the (latent) causal processes.
Note that invertible component-wise transformations onlatent causal processes do not change their conditional independence relations.
Definition 1 (Identifiable Latent Causal Processes) .
Formally let { x t } Tt =1 be a sequence of observedvariables generated by the true temporally causal latent processes specified by ( f i , θ r , p ( ϵ i ) , g )given in Eq.
A learned generative model ( ˆ f i , ˆ θ r , ˆ p ( ϵ i ) , ˆ g ) is observationally equivalentto ( f i , θ r , p ( ϵ i ) , g ) if the model distribution p ˆ f, ˆ θ r , ˆ p ϵ , ˆ g ( { x t } Tt =1 ) matches the data distributionp f, θ r ,p ϵ , g ( { x t } Tt =1 ) everywhere.
We say latent causal processes are identifiable if observational 175 equivalence can lead to identifiability of the latent variables up to permutation π and component-wiseinvertible transformation T :
[Equation]
where X is the observation space.
Identifiability Theory
We establish the identifiability theory of nonparametric time-delayed latent causal processes underthree different types of distribution shifts.
W.l.o.g.
, we consider the latent processes with maximumtime lag L = 1 .
Extentions to arbitrary time lags have been discussed in Appendix S1.5.
Let kbe the element index of the latent space z t and the latent size is n .
In particular, (1) under fixed causal dynamics, we leverage the distribution changes p ( z k,t | z t − 1 ) for different values of z t − 1 ; (2)under changing causal dynamics, we exploit the changing causal influences on p ( z k,t | z t − 1 , u r ) underdifferent domain u r , and (3) under global observation changes, the nonstationarity p ( z k,t | u r ) underdifferent values of u r is exploited.
The proofs are provided in Appendix S1.
The comparisonsbetween existing theories are in Appendix S1.3.
4.1 Identifiability under Fixed Causal Dynamics
Let η kt ≜ log p ( z k,t | z t − 1 ) .
Assume that η kt is twice differentiable in z k,t and is differentiable inz
l,t − 1 , l = 1 , 2 , ..., n .
Note that the parents of z k,t may be only a subset of z t − 1 ; if z l,t − 1 is not aparent of z k,t , then ∂η kt ∂z l,t − 1 = 0 .
Below we provide a sufficient condition for the identifiability of z t ,followed by a discussion of specific unidentifiable and identifiable cases to illustrate how general it is.
Theorem 1 (Identifiablity under a Fixed Temporal Causal Model) .
Suppose there exists invertiblefunction ˆ g that maps x t to ˆ z t , i.e.,
[Equation]
(3) such that the components of ˆ z t are mutually independent conditional on ˆ z t − 1 .
Let
[Equation]
..., ∂ 3 η kt ∂z 2 k,t ∂z n,t − 1 (cid:17) ⊺ .
If for each value of z t , v 1 ,t , ˚ v 1 ,t , v 2 ,t , ˚ v 2 ,t , ..., v n,t , ˚ v n,t , as 2 n vector functions in z 1 ,t − 1 , z 2 ,t − 1 , ..., 197 z n,t − 1 , are linearly independent, then z t must be an invertible, component-wise transformation of apermuted version of ˆ z t .
The linear independence condition in Theorem 1 is the core condition to guarantee the identifiabilityof z t from the observed x t .
To make this condition more intuitive, below we consider specificunidentifiable cases, in which there is no temporal dependence in z t or the noise terms in z t areadditive Gaussian, and two identifiable cases, in which z t has additive, heterogeneous noise or followssome linear, non-Gaussian temporal process.
Let us start with two unidentifiable cases.
In case N 1 , t t is an independent and identically distributed(i.i.d.)
process, i.e., there is no causal influence from any component of z t − 1 to any z k,t .
In thiscase, v k,t and ˚ v k,t (defined in Eq. 4) are always 0 for k = 1 , 2 , ..., n , since p ( z k,t | z t − 1 ) does notinvolve z t − 1 .
So the linear independence condition is violated.
In fact, this is the regular nonlinear
ICA problem with i.i.d.
data, and it is well-known that the underlying independent variables are notidentifiable [5].
In case N 2 , all z k,t follow an additive noise model with Gaussian noise terms, i.e.,
[Equation]
where q is a transformation and the components of the Gaussian vector ϵ t are independent andalso independent from z t − 1 .
Then ∂ 2 η kt ∂z 2 k,t is constant, and ∂ 3 η kt ∂z 2 k,t ∂z l,t − 1 ≡ 0 , violating the linearindependence condition.
In the following proposition we give some alternative solutions and verifythe unidentifiability in this case.
Proposition 1 (Unidentifiability under Gaussian Noise) .
Suppose x t = g
( z t ) was generated by
Eq.
5, where the components of ϵ t are mutually independent Gaussian and also independent fromz t − 1 .
Then any ˆ z t =
D 1 UD 2 · z t , where D 1 is an arbitrary non-singular diagonal matrix, U is anarbitrary orthogonal matrix, and D 2 is a diagonal matrix with V ar − 1 / 2 ( ϵ k,t ) as its k th diagonalentry, is a valid solution to satisfy the condition that the components of ˆ z t are mutually independentconditional on ˆ z t − 1 .
Roughly speaking, for a randomly chosen conditional density function p ( z k,t | z t − 1 ) in which z k,tis not independent from z t − 1 (i.e., there is temporal dependence in the latent processes) and whichdoes not follow an additive noise model with Gaussian noise, the chance for its specific second- andthird-order partial derivatives to be linearly dependent is slim.
Now let us consider two cases inwhich the latent temporally processes z t are naturally identifiable.
First, consider case Y 1 , where z k,tfollows a heterogeneous noise process, in which the noise variance depends on its parents:
[Equation]
Here we assume ϵ k,t is standard Gaussian and ϵ 1 ,t , ϵ 2 ,t , .., ϵ n,t are mutually independent and independent from z t − 1 .
1 b k , which depends on z t − 1 , is the standard deviation of the noise in z k,t .
(Forconciseness, we drop the argument of b k and q k when there is no confusion.)
Note that in this model,if q k is 0 for all k = 1 , 2 , ..., n , it reduces to a multiplicative noise model.
The identifiability result ofz t is established in the following corollary.
Corollary 1 (Identifiability under Heterogeneous Noise) .
Suppose x t = g
( z t ) was generatedaccording to Eq. 6, and Eq.
3 holds true.
If b k · ∂b k ∂ z t − 1 and b k · ∂b k ∂ z t − 1 ( z k,t − q k ) − b 2 k · ∂q k ∂ z t − 1 , withk = 1 , 2 , ..., n , which are in total 2 n function vectors in z t − 1 , are linearly independent, then z t mustbe an invertible, component-wise transformation of a permuted version of ˆ z t .
Let us then consider another special case, denoted by Y 2 , with a linear, non-Gaussian temporal modelfor z t : the latent processes follow Eq. 5, with q being a linear transformation and ϵ k,t following aparticular class of non-Gaussian distributions.
The following corollary shows that z t is identifiable aslong as each z k,t receives causal influences from some components of z t − 1 .
Corollary 2 (Identifiability under a Specific Linear, Non-Gaussian Model for Latent Processes) .
Suppose x t = g
( z t ) was generated according to Eq.
5, in which q is a linear transformation andfor each z k,t , there exists at least one k ′ such that c k,k ′ ≜ ∂z k,t ∂z k ′ ,t − 1 ̸ = 0 .
Assume the noise term ϵ k,tfollows a zero-mean generalized normal distribution:
[Equation]
If Eq. 3 holds, then z t must be an invertible, component-wise transformation of permuted ˆ z t .
4.2 Further Benefits from Changing Causal Influences
Let η kt ( u r ) ≜ log p ( z k,t | z t − 1 , u r ) where r = 1 , ..., m .
[14] established the identifiability of thelatent temporal causal processes z t in certain nonstationary cases, under the condition that thenoise term in each z k,t , relative to its parents in z t − 1 , changes across m contexts correspondingto u = u 1 , u 2 , ..., u m .
Here we show that the identifiability result shown in the previous sectioncan further benefit from nonstationarity of the causal model, and that our identifiability condition isgenerally much weaker than that in [14]: we allow changes in the noise term or causal influence onz k,t from its parents in z t − 1 , and our “sufficient variability" condition is just a necessary conditionfor that in [14] because of the additional information that one can leverage.
Let v k,t ( u r ) be v k,t ,which is defined in Eq.
4, in the u r context.
Similarly, Let ˚ v k,t ( u r ) be ˚ v k,t in the u r context.
Let
[Equation]
where ∆ 2 r = ∂ 2 η kt ( u r ) ∂z 2 k,t − ∂ 2 η kt ( u r − 1 ) ∂z 2 k,t and ∆ r = ∂η kt ( u r ) ∂z k,t − ∂η kt
( u r − 1 )
∂z k,t .
As provided below, in ourcase, the identifiablity of z t is guaranteed by the linear independence of the whole function vectorss k,t and ˚ s k,t , with k = 1 , 2 , ..., n .
However, the identifiability result in [14] relies on the linearindependence of only the last m − 1 components of s k,t and ˚ s k,t with k = 1 , 2 , ..., n ; this linearindependence is generally a much stronger and restricted condition.
Theorem 2 (Identifiability under Changing Causal Dynamics) .
Suppose x t = g
( z t ) and that theconditional distribution p ( z k,t | z t − 1 ) may change across m values of the context variable u , denotedby u 1 , u 2 , ..., u m .
Suppose the components of z t are mutually independent conditional on z t − 1 ineach context.
Assume that the components of ˆ z t produced by Eq.
3 are also mutually independentconditional on ˆ z t − 1 .
If the 2 n function vectors s k,t and ˚ s k,t , with k = 1 , 2 , ..., n , are linearlyindependent, then ˆ z t is a permuted invertible component-wise transformation of z t .
Theorem 3 (Identifiability under Observation Changes) .
Suppose x t = g
( z t ) and that the conditionaldistribution p ( z k,t | u ) may change across m values of the context variable u , denoted by u 1 , u 2 ,..., u m .
Suppose the components of z t are mutually independent conditional on u in each context.
Assume that the components of ˆ z t produced by Eq.
3 are also mutually independent conditional onˆ z t − 1 .
If the 2 n function vectors s k,t and ˚ s k,t , with k = 1 , 2 , ..., n , are linearly independent, then ˆ z tis a permuted invertible component-wise transformation of z t .
Corollary 3 (Identifiability under Modular Distribution Shifts) .
Assume the data generating processin Eq. 1.
If the three partitioned latent components z t = ( z fix t , z chg t , z obs t ) respectively satisfy theconditions in Theorem 1, Theorem 2, and Theorem 3, then z t must be an invertible, component-wisetransformation of a permuted version of ˆ z t .
C. Change Factor Representation
B. Factorized Inference
Element / Channel Index 𝑧̂ #,% 𝑧̂ &,%’( 𝑧̂ ),%’( … 𝑧̂ #*(,% 𝑧̂ &,%’+ 𝑧̂ ),%’+ … … … … …
Time Index … 𝐳-… 𝜇 0 … 𝐳-
𝐋 𝐳- 𝐓 … 𝐱-…
𝐱-
𝐋 𝐱-
𝐓 𝐱… 𝐱 𝐋 𝐱 𝐓 …
A. Enc.
A. Enc. A. Enc.
𝜎𝜇 + 𝜎 + 𝜇 ( 𝜎 (
E. Dec E. Dec
E. Dec 𝑧̂ 6,% 𝑧̂ &,%’( 𝑧̂ ),%’( … 𝑧̂ 6*(,% 𝑧̂ &,%’+ 𝑧̂ ),%’+ … … … … … 𝑧̂ 7,% 𝑧̂ 7*(,% … 𝐮 𝐫
[Figure]
𝐫𝐝𝐲𝐧
Embedding Layer ε- @,A ε-
B,A 𝛉; 𝐫𝐨𝐛𝐬 𝐟 𝐜 (𝛉; 𝐫𝐝𝐲𝐧 ) 𝐟 𝐨 (𝛉; 𝐫𝐨𝐛𝐬 )
D. Modular Prior Network
Fixed 𝐟 𝐬 Time-delayed information
[Figure]
Our Approach
5.1 LCD-NM: Learning Latent Causal Dynamics from Nonlinear Mixtures
Given our identifiability results, we propose LCD-NM framework to estimate the latent causaldynamics under modular distribution shifts, by extending Sequential Variational Auto-Encoders [31]with tailored modules to model different distribution shifts, and enforcing the conditions in Sec.
4as constraints.
We give the estimation procedure of the latent causal dynamics model in Eq. 1.
Themodel architecture is showcased in Fig.
The framework has the following three major components.
The implementation details are in Appendix S3.3.
Specifically, we leverage the partitioned estimatedlatent subspaces ˆ z t =
(ˆ z fix t , ˆ z chg t , ˆ z obs t ) and model their distribution changes in conditional transitionpriors.
We use [ f s , f c , f o ] to capture causal relations as in Eq. 1, where f s captures fixed causalinfluences, f c for changing causal influences, and f o for observation changes.
Accordingly, we learn[ ˆ f − 1 s , ˆ f − 1 c , ˆ f − 1 o ] to output random process noise from the estimated direct cause (lagged states ˆ z Hx )and effect (current states) variables ˆ z t .
Change Factor Representation We learn to embed domain index u r into low-dimensional changefactors (ˆ θ dyn r , ˆ θ obs r ) in Fig.
2C and insert them as external inputs to the (inverse) dynamics functionˆ f − 1 c (ˆ θ dyn r ) , or the observation bijector ˆ f − 1 o
(ˆ θ obs r ) , respectively.
And hence the distribution shifts arecaptured and utilized in the implementation.
Modular Prior Network We follow standard conditional normalizing flow formulation [32, 14].
Let z Hx denote the lagged latent variables up to maximum time lag L .
In particular, for 1) fixedcausal dynamics processes ˆ z fix t , their transition priors are obtained by first learning inverse transitionfunctions f − 1 s that take the estimated latent variables and output random noise terms, and applyingthe change of variables formula to the transformation: p (ˆ z fix s,t | ˆ z Hx ) =
p ϵ s (cid:16) ˆ f − 1 s (ˆ z fix s,t , ˆ z Hx ) (cid:17) (cid:12)(cid:12)(cid:12) ∂ ˆ f − 1 s ∂ ˆ z fix s,t (cid:12)(cid:12)(cid:12) ;for 2) changing causal dynamics , we evaluate p (ˆ z chg c,t | ˆ z Hx , u r ) =
p ϵ c (cid:16) ˆ f
[Table] [Table]
[Table] [Table]
, ˆ
[Table]
) (cid:17) (cid:12)(cid:12)(cid:12)
[Table]
[Table]
(cid:12)(cid:12)(cid:12)by learning a holistic inverse dynamics ˆ f − 1 c that takes the estimated change factors for dynamicsˆ θ dyn r as inputs, and similarly for 3) observation changes ˆ z obs t , we learn to project them to invariantnoise terms by ˆ f − 1 o which takes the change factors θ obs r as arguments, and obtains p (ˆ z obs o,t | u r ) =
p
ϵ
o (cid:16) ˆ f − 1 o
(ˆ z obs o,t , ˆ θ obs r ) (cid:17) (cid:12)(cid:12)(cid:12) ∂ ˆ f − 1 o ∂
ˆ z obs o,t (cid:12)(cid:12)(cid:12) as the prior.
Conditional independence of the estimated latentvariables p (ˆ z t | ˆ z Hx ) is enforced by summing up all estimated component densities when obtainingthe joint p ( z t | z
Hx , u ) in Eq. 9.
Given that the Jacobian is lower-triangular, we can compute itsdeterminant as the product of diagonal terms.
The detailed derivations are given in Appendix S3.1.
log p (ˆ z t | ˆ z
Hx , u r ) =
n (cid:88)
i = log p (ˆ ϵ i | u r ) (cid:124) (cid:123)(cid:122) (cid:125)
Conditional indepdence + n (cid:88)
i = log (cid:12)(cid:12)(cid:12) ∂
ˆ f − 1 i ∂
ˆ z it (cid:12)(cid:12)(cid:12) (cid:124) (cid:123)(cid:122) (cid:125) Low-triangular Jacobian matrix (9)
Factorized Inference
We infer the posteriors of each time step q (ˆ z t | x t ) using only the observationat that time step, because in Eq.
1, x t preserves all the information of the current system states sothe joint probability q ( ˆz 1: T | x 1: T ) can be factorized into product of these terms.
We approximate theposterior q ( ˆz t | x t ) with an isotropic Gaussian with mean and variance from the inference network.
Optimization We train LCD-NM using the ELBO objective L ELBO = 1 N (cid:80) i ∈
N L Recon − β L KLD , inwhich we use mean-squared error (MSE) for the reconstruction likelihood L Recon .
The KL divergence
L KLD is estimated via a sampling approach since with a learned nonparametric modular transitionprior, the distribution does not have an explicit form.
Specifically, we obtain the log-likelihood ofthe posterior, evaluate the prior – log p (ˆ z t | ˆ z Hx , u r ) in Eq. 9, and compute their mean difference in abatch B as the KL loss: L KLD =
(cid:80) i ∈B E ˆz t ∼ q (cid:16) ˆz ( i ) t | x ( i ) t (cid:17) log q ( ˆz t | x t ) − log p (ˆ z t | ˆ z Hx , u r ) .
5.2 Causal Visualization
For visualization purposes, when the underlying latent processes have sparse causal relations, wefit LassoNet [33] on the latent processes recovered by LCD-NM to interpret the causal relations.
Specifically, we fit LassoNet to predict ˆ z t using the estimated history information ˆ z Hx = { ˆ z t − τ } Lτ =1up to maximum time lag L .
Note that this postprocessing step is optional – the latent causal relationshave already been captured in the learned transition functions in LCD-NM.
Also, our identifiabilityconditions do not rely on the sparsity of causal relations in the latent processes.
Experimental Results
We evaluate the identifiability results of LCD-NM on a number of simulated and real-world timeseries datasets.
We first introduce the evaluation metrics and baselines.
(1) Evaluation Metrics.
Toevaluate the identifiability of the latent variables, we compute Mean Correlation Coefficient (MCC)on the test dataset.
MCC is a standard metric in the ICA literature for continuous variables whichmeasure the identifiability of the learned latent causal processes.
MCC is close to 1 when latentvariables are identifiable up to permutation and component-wise invertible transformation in thenoiseless case.
(2) Baselines.
Nonlinear ICA methods are used: (1) BetaVAE [34] which ignores bothhistory and nonstationarity information; (2) iVAE [9] and TCL [6] which leverage nonstationarity toestablish identifiability but assumes independent factors, and (3) SlowVAE
[12] and PCL [7] whichexploit temporal constraints but assume independent sources and stationary processes, and (4) LEAP[14] which assumes nonstationary, causal processes but only models nonstationary noise.
Two otherdisentangled deep state-space models with nonlinear dynamics models: Kalman VAE (KVAE [35])and Deep Variational Bayes Filters (DVBF [36]), are also used for comparisons.
[Table]
6.1 Simulated Experiments
We generate synthetic datasets that satisfy our identifiability conditions in the theorems followingthe procedures in Appendix S2.1.1.
As in Table 2, our framework can recover the latent processesunder fixed dynamics (heterogeneous noise model), under changing causal dynamics, and undermodular distribution shifts with high MCCs (>0.95).
The baselines that do not exploit history (i.e.,β VAE, i-VAE, TCL), with independent source assumptions (SlowVAE, PCL), considers limitednonstationary cases (LEAP) distort the identifiability results.
KVAE and DVBF achieve MCCs (0.8)under fixed dynamics but distorts the identifiability under changing dynamics and modular shiftsetting because they don’t model changing causal relations and observation changes.
6.2 Real-world Applications
Video Data –
Modified Cartpole Environment We evaluate LCD-NM on the modified cartpole[30] video dataset and compare the performances with the baselines.
Modified Cartpole is a nonlineardynamical system with cart positions x t and pole angles θ t as the true state variables.
The datasetdescriptions are in Appendix S2.1.2.
We use 6 source domains with different gravity values g ={ 5 , 10 , 15 , 20 , 25 , 30 } .
Together with the 2 discrete actions (i.e., left and right), we have 12 segmentsof data with changing causal dynamics.
We fit LCD-NM with two-dimensional change factors θ dyn r .
We set the latent size n = 8 and the lag number L = 2 .
In Fig.
3, the latent causal processes arerecovered, as seen from (a) high MCC for the latent causal processes; (b) the latent factors areestimated up to component-wise transformation; (c) LCD-NM outperforms the baselines and (d) thelatent traversals confirm the two recovered latent variables correspond to the position and pole angle.
[Figure]
Motion Capture Data – CMU-Mocap
We experimented with another real-world motion capturedataset (CMU-Mocap).
The dataset descriptions are in Appendix S2.1.2.
We fit LCD-NM with 11trials of motion capture data for subject #8.
The 11 trials contain walk cycles with very different styles(e.g.
, slow walk, stride).
We set latent size n = 3 and lag number L = 2 .
The differences betweentrials are captured through learning the 2-dimensional change factors for each trial.
In Fig. 4(a), thelearned change factors group similar walk styles into clusters; in Panel (c), three latent variables(which seem to be pitch, yaw, roll rotations) are found to explain most of the variances of humanwalk cycles.
The learned latent coordinates (Panel b) show smooth cyclic patterns with differencesamong different walking styles.
For the discovered skeleton (Panel d), roll and pitch of walking arefound to be causally-related while yaw has independent dynamics.
[Figure]
Conclusion
In this paper, without relying on parametric or distribution assumptions, we established identifiabilitytheories for nonparametric latent causal processes from their observed nonlinear mixtures in stationaryenvironments and under unknown distribution shifts.
The basic limitation of this work is that theunderlying latent processes are assumed to have no instantaneous causal relations but only timedelayed influences.
If the time resolution of the observed time series is the same as the causalfrequency (or even higher), this assumption naturally holds.
However, if the resolution of the timeseries is much lower, then it is usually violated and one has to find a way to deal with instantaneouscausal relations.
On the other hand, it is worth mentioning that the assumptions are generally testable;if the assumptions are actually violated, one can see that the results produced by our method may notbe reliable.
Extending our theories and framework to address the issue of instantaneous dependencyor instantaneous causal relations will be one line of our future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
If you used crowdsourcing or conducted research with human subjects...
Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Supplement to
534 “Causal
Disentanglement for Time Series”
We divide the supplement into four sections.
In S1, we give the proofs of the theorems and compare ourtheory with the existing nonlinear ICA work in terms of mathematical formulation and assumptions.
In S2, we provide the descriptions of the datasets used in this paper and evaluation metrics.
In S3, wefirst derive the computation of the modular prior likelihood.
The model architecture, hyperparameterselection and training are also explained in detail.
S1 Identifiability Theory
S1.1 Proof for Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S1.2 Proof for Theorem 2 and 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S1.3 Comparisons with Existing Nonlinear ICA Theories . .
S1.4 Discussion of the Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S1.4.1 Linear Independence Condition . . . . . . . . . . . . . . . . . . . . . . .
S1.4.2 Independent Noise Condition . . . . . . . . . . . . . . . . . . . . . . . .
S1.4.3 Causal Influences between Observed Variables . . . . . . . . . . . . . . .
S1.5 Extension to Multiple Time Lags . . . . . . . . . . . . . . . . . . . . . . . . . . .
S2 Experiment Settings
S2.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S2.1.1 Synthetic Dataset Generation . . . . . . . . . . . . . . . . . . . . . . . . .
S2.1.2 Real-world Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S2.2 Mean Correlation Coefficient . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S3 Implementation Details
S3.1 Modular
Prior Likelihood Derivation . . . . . . . . . . . . . . . . . . . . . . . . .
S3.2 Comparisons with AdaRL [30] . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S3.3 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S3.4 Hyperparameter and Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S4 Additional Experiment Results
S Identifiability Theory
The observed variables were generated according to :x t =
g ( z t ) , (10) in which g is invertible, and z it , as the i th component of z t , is generated by (some) components ofz t − 1 and noise E it .
E 1 t , E 2 t , ..., E nt are mutually independent.
In other words, the components ofz t are mutually independent conditional on z t − 1 .
Let η kt ≜ log p ( z kt | z t − 1 ) .
Assume that η k ( t ) istwice differentiable in z kt and is differentiable in z l,t − 1 , l = 1 , 2 , ..., n .
Note that the parents of z ktmay be only a subset of z t − 1 ; if z l,t − 1 is not a parent of z kt , then ∂η k ∂z l,t − 1 = 0 .
S1.1 Proof
for Theorem
Theorem S1 (Identifiablity under a Fixed Temporal Causal Model) .
Suppose there exists invertiblefunction f , which is the estimated mixing function (i.e., we use f and ˆ g interchangeably in Appendix) that maps x t to ˆ z t , i.e.,
[Equation]
x
[Equation]
such that the components of ˆ z t are mutually independent conditional on ˆ z t − 1 .
Let
[Equation]
If for each value of z t , v 1 t , ˚ v 1 t , v 2 t , ˚ v 2 t , ..., v nt , ˚ v nt , as 2 n vector functions in z 1 ,t − 1 , z 2 ,t − 1 , ...,z n,t − 1 , are linearly independent, then z t must be an invertible, component-wise transformation of apermuted version of ˆ z t .
Proof.
Combining Eq.
10 and Eq.
11 gives z t = g − 1 ( f − 1 (ˆ z t ))
=
h (ˆ z t ) , where h ≜ g − 1 ◦ f − 1 .
Since both f and g are invertible, h is invertible.
Let H t be the Jacobian matrix of the transformationh (ˆ z t ) , and denote by H kit its ( k, i ) th entry.
First, it is straightforward to see that if the components of ˆ z t are mutually independent conditional on ˆ z t − 1 , then for any i ̸
= j , ˆ z it
and ˆ z jt are conditionally independent given ˆ z t − 1 ∪ (ˆ z t \ { ˆ z it , ˆ z jt } ) .
Mutual independence of the components of ˆ z t conditional on ˆ z t − 1 implies that ˆ z it is independent from ˆ z t \ { ˆ z it ,
ˆ z jt } conditional on ˆ z t − 1 , i.e.,
[Equation]
At the same time, it also implies ˆ z
it is independent from ˆ z t \ { ˆ z it } conditional on ˆ z t − 1 , i.e.,
[Equation]
Combining the above two equations gives p (ˆ z it | ˆ z t − 1 ∪ (ˆ z t \{ ˆ z it } ))
=
p (ˆ z it | ˆ z t − 1 ∪ (ˆ z t \{ ˆ z it , ˆ z jt } )) ,i.e., for i ̸
= j , ˆ z it
and ˆ z jt are conditionally independent given ˆ z t − 1 ∪ (ˆ z t \ { ˆ z it , ˆ z jt } ) .
We then make use of the fact that if ˆ z it and ˆ z jt are conditionally independent given ˆ z t − 1 ∪ (ˆ z t \ { ˆ z it , ˆ z jt } ) , then
[Equation]
assuming the cross second-order derivative exists [37].
Since p (ˆ z t , ˆ z t − 1 ) = p (ˆ z t | ˆ z t − 1 ) p (ˆ z t − 1 )while p (ˆ z t − 1 ) does not involve ˆ z it or ˆ z
jt , the above equality is equivalent to
[Equation]
The Jacobian matrix of the mapping from ( x t − 1 , ˆ z t ) to ( x t − 1 , z t ) is (cid:20) I ∗ H t (cid:21) , where ∗ stands fora matrix, and the (absolute value of the) determinant of this Jacobian matrix is | H t | .
Thereforep (ˆ z t , x t − 1 ) = p ( z t , x t − 1 ) · | H t | .
Dividing both sides of this equation by p ( x t − 1 ) gives
[Equation]
Since p ( z t | z t − 1 ) =
p ( z t | g ( z t − 1 ))
=
p ( z t | x t − 1 ) and similarly p (ˆ z t | ˆ z t − 1 ) =
p (ˆ z t | x t − 1 ) , Eq.
14tells us
[Equation]
Its partial derivative w.r.t.
ˆ z it is
[Equation]
Its second-order cross derivative is
[Equation]
The above quantity is always 0 according to Eq. 13.
Therefore, for each l = 1 , 2 , ..., n and each valuez l,t − 1 , its partial derivative w.r.t.
z l,t − 1 is always 0.
That is,
[Equation]
where we have made use of the fact that entries of H t do not depend on z l,t − 1 .
If for any value of z t , v 1 t , ˚ v 1 t , v 2 t , ˚ v 2 t , ..., v nt , ˚ v nt are linearly independent, to make the aboveequation hold true, one has to set H kit H kjt
= 0 or i ̸
= j .
That is, in each row of H t there is only onenon-zero entry.
Since h is invertible, then z t must be an invertible, component-wise transformationof a permuted version of ˆ z t .
The linear independence condition in Theorem S1 is the core condition to guarantee the identifiabilityof z t from the observed x t .
Roughly speaking, for a randomly chosen conditional density functionp ( z kt | z t − 1 ) , the chance for this constraint to hold on its second- and third-order partial derivatives isslim.
For illustrative purposes, below we make this claim more precise, by considering a specificunidentifiable case, in which the noise terms in z t are additive Gaussian, and two identifiable cases,in which z t has additive, heterogeneous noise or follows some linear, non-Gaussian temporal process.
Let us start with an unidentifiable case.
If all z kt follow the additive noise model with Gaussian noiseterms, i.e.,
[Equation]
where q is a transformation and the components of the Gaussian vector E t are independent andalso independent from z t − 1 .
Then ∂ 2 η kt ∂z 2 kt is constant, and
[Equation]
0 , violating the linearindependence condition.
In the following proposition we give some alternative solutions and verifythe unidentifiability in this case.
Proposition S1 (Unidentifiability under Gaussian noise) .
Suppose x t was generated accordingto Eq.
10 and Eq. 18, where the components of E t are mutually independent Gaussian and alsoindependent from z t − 1 .
Then any ˆ z t =
D 1 UD 2 · z t , where D 1 is an arbitrary non-singular diagonalmatrix, U is an arbitrary orthogonal matrix, and D 2 is a diagonal matrix with V ar − 1 / 2 ( E kt ) as itsk th diagonal entry, is a valid solution to satisfy the condition that the components of ˆ z t are mutuallyindependent conditional on ˆ z t − 1 .
Proof.
In this case we have
[Equation]
It is easy to verify that the components of D 1 UD 2 · E t are mutually independent and are independentfrom D 1 UD 2 · q ( z t − 1 ) .
As a consequence, ˆ z t are mutually independent conditional on ˆ z t − 1 .
Now let us consider some cases in which the latent temporally processes z t are naturally identifiableunder some technical conditions.
Let us first consider the case where z kt follows a heterogeneousnoise process, in which the noise variance depends on its parents:
[Equation]
Here we assume E kt is standard Gaussian and E 1 t , E 2 t , .., E nt are mutually independent and independent from z t − 1 .
1 b k , which depends on z t − 1 , is the standard deviation of the noise in z kt .
(Forconciseness, we drop the argument of b k and q k when there is no confusion.)
Note that in this model,if q k is 0 for all k = 1 , 2 , ..., n , it reduces to a multiplicative noise model.
The identifiability result ofz t is established in the following proposition.
Corollary S1 (Identifiablity under Heterogeneous Noise) .
Suppose x t was generated according to
Eq.
10 and Eq. 19.
Suppose Eq.
11 holds true.
If b k · ∂b k ∂ z t − 1 and b k · ∂b k ∂ z t − 1 ( z kt − q k ) − b 2 k · ∂q k ∂ z t − 1 ,with k = 1 , 2 , ..., n , which are in total 2 n function vectors in z t − 1 , are linearly independent, then z tmust be an invertible, component-wise transformation of a permuted version of ˆ z t .
Proof.
Under the assumptions, one can see that
[Equation]
Consequently, one can find
[Equation]
Then the linear independence of v kt and ˚ v kt (defined in Eq.
12), with k = 1 ., 2 , ..., n , reduces tothe linear independence condition in this proposition.
Theorem S1 then implies that z t must be aninvertible, component-wise transformation of a permuted version of ˆ z t .
Let us then consider another special case, with linear, non-Gaussian temporal model for z t : the latentprocesses follow Eq. 18, with q being a linear transformation and E kt following a particular class ofnon-Gaussian distributions.
The following corollary shows that z t is identifiable as long as each z ktreceives causal influences from some components of z t − 1 .
Corollary S2 (Identifiablity under a Specific Linear, Non-Gaussian Model for Latent Processes) .
Suppose x t was generated according to Eq.
10 and Eq.
18, in which q is a linear transformation andfor each z kt , there exists at least one k ′ such that c kk ′ ≜
∂z kt ∂z k ′ ,t − 1 ̸ = 0 .
Assume the noise term E ktfollows a zero-mean generalized normal distribution:
[Equation]
Suppose Eq.
11 holds true.
Then z t must be an invertible, component-wise transformation of apermuted version of ˆ z t .
Proof.
In this case, we have
[Equation]
We know that | e lt | β − 2 and | e lt | β − 3 are linearly independent (because their ratio, | e lt | , is not constant).
Furthermore, | e lt | β − 2 and | e lt | β − 3 , with l = 1 , 2 , ..., n , are 2 n linearly independent functions(because of the different arguments involved).
Suppose there exist α l
and α l 2 , with l = 1 , 2 , ..., n , such that
[Equation]
[Equation]
It is assumed that for each k = 1 , 2 , ..., n , there exists at least one k ′ such that c kk ′ ̸
= 0 .
Eq.
22 thenimplies that for any k we have
[Equation]
Since | e lt | β − 2 and | e lt | β − 3 , with l = 1 , 2 , ..., n , are linearly independent and c kk ′ ̸ = 0 , to make theabove equation hold, one has to set α k
= α k 2 = 0 .
As this applies to any k , we know that for Eq.
22to be satisfied, α l 1 and α l 2 must be 0, for all l = 1 , 2 , ..., n .
That is, v 1 t , ˚ v 1 t , v 2 t , ˚ v 2 t , ..., v nt , ˚ v ntare linearly independent.
The linear independence condition in Theorem S1 is satisfied.
Therefore z tmust be an invertible, component-wise transformation of a permuted version of ˆ z t .
S1.2 Proof for Theorem 2 and
Let v kt ( u r ) be v kt , which is defined in Eq.
12, in the u r context.
Similarly, Let ˚ v kt ( u r ) be ˚ v kt in the u r context.
Let
[Equation]
As provided below, in our case, the identifiablity of z t is guaranteed by the linear independence ofthe whole function vectors s kt and ˚ s kt , with k = 1 , 2 , ..., n .
However, the identifiability result in Yaoet al.
(2021) relies on the linear independence of only the last m − 1 components of s kt and ˚ s kt withk = 1 , 2 , ..., n ; this linear independence is generally a much stronger condition.
Theorem S2 (Identifiability under Changing Causal Dynamics) .
Suppose the observed processesx t was generated by Eq.
10 and that the conditional distribution p ( z kt | z t − 1 ) may change acrossm values of the context variable u , denoted by u 1 , u 2 , ..., u m .
Suppose the components of z t aremutually independent conditional on z − 1 in each context.
Assume that the components of ˆ z t producedby Eq.
11 are also mutually independent conditional on ˆ z t − 1 .
If the 2 n function vectors s kt and˚ s kt , with k = 1 , 2 , ..., n , are linearly independent, then ˆ z t is a permuted invertible component-wisetransformation of z t .
Proof.
As in the proof of Theorem S1, because the components of ˆ z t are mutually independentconditional on ˆ z t − 1 , we know that for i ̸
= j ,
[Equation]
Compared to Eq.
16, here we allow p (ˆ z t | ˆ z t − 1 ) to depend on u .
Since the above equation is always0, taking its partial derivative w.r.t.
z l,t − 1 gives
[Equation]
Similarly, Using different values for u in Eq. 24 take the difference of this equation across them gives
[Equation]
Therefore, if s kt and ˚ s kt , for k = 1 , 2 , ..., n , are linearly independent, H kit H kjt has to be zero forall k and i ̸
= j .
Then as shown in the proof of Theorem S1, ˆ z t must be a permuted component-wiseinvertible transformation of z t .
Theorem S3 (Identifiability under Observation Changes) .
Suppose x t = g
( z t ) and that the conditional distribution p ( z k,t | u ) may change across m values of the context variable u , denoted by u 1 ,u 2 , ..., u m .
Suppose the components of z t are mutually independent conditional on u in each context.
Assume that the components of ˆ z t produced by Eq.
3 are also mutually independent conditional onˆ z t − 1 .
If the 2 n function vectors s k,t and ˚ s k,t , with k = 1 , 2 , ..., n , are linearly independent, then ˆ z tis a permuted invertible component-wise transformation of z t .
Proof.
As in the proof of Theorem S2, because z t is not dependent on the history z t − 1 so are thecomponents of ˆ z t , the conditioning on ˆ z t in Eq.
24 and the following equations can be removedbecause of the independence.
This directly leads to the same conclusion as in Theorem S2.
Corollary S3 (Identifiability under Modular Distribution Shifts) .
Assume the data generating processin Eq. 1.
If the three partitioned latent components z t = ( z fix t , z chg t , z obs t ) respectively satisfy theconditions in Theorem 1, Theorem 2, and Theorem 3, then z t must be an invertible, component-wisetransformation of a permuted version of ˆ z t .
Proof.
Because the three partitioned subspaces ( z fix t , z chg t , z obs t ) are conditional independent given thehistory and domain index, it is straightforward to factorize the joint conditional log density into threecomponents.
By using the proof in Theorem 1, 2, and 3, we can directly derive the same quantity asin Eq.
15 or Eq.
Therefore, if s kt and ˚ s kt , for k = 1 , 2 , ..., n , are linearly independent, H kit H kjthas to be zero for all k
and i ̸
= j .
Then as shown in the proof of Theorem S1, ˆ z t must be a permutedcomponent-wise invertible transformation of z t .
S1.3 Comparisons with Existing Nonlinear ICA Theories
We compare our established theory with (1) PCL [7], (2) SlowVAE [12], (3) i-VAE [9], (4)
GCL [8]and (5) LEAP [14] in terms of their mathematical formulation and assumptions.
PCL [7] The formulation of the underlying processes in PCL is in Eq.
[Equation]
or
[Equation]
27) where G is a non-quadratic function corresponding to the log-pdf of innovations, ρ < 1 is theregression coefficient, r is some nonlinear, strictly monotonic regression, and λ is a positive precisionparameter.
PCL is applicable to stationary environments only when the sources z it are mutually independent(see Assumption 1 of Theorem 1 in PCL) and follow functional and distribution assumptions in
Eq.
Our formulation allows latent variables to have arbitrary, nonparametric time-delayed causalrelations in between without functional form or distribution assumptions.
SlowVAE [12] The formulation of the underlying sources in SlowVAE is in Eq.
[Equation]
where the latent processes have independent, identity transitions with generalized Laplacian noises.
SlowVAE established identifiability under stationary, mutually independent processes (similar to
PCL [7]), in which time-delayed causal influences are not allowed.
Furthermore, it assumes thatthe transition function of each independent process is an identity function and the process noise hasgeneralized Laplacian distribution.
Our Theorem 1 includes both SlowVAE and PCL as specialcases, in the sense that (1) we remove the functional and distributional assumptions to allow the latentprocesses to have nonparametric causal influences in between, and (2) our Corollay 2 , which is anillustrative example of Theorem 1 , further completes to Eq.
by allowing linear time-delayedtransitions in the latent process with non-Gaussian noises .i-VAE [9] Similar to TCL [6] and GIN [10], i-VAE exploits the nonstationarity brought by classlabels on the distribution of latent variables.
As one can see from Eq.
29, the latent variables areconditionally independent, without causal relations in between while all of our theorems consider(time-delayed) causal relations between latent variables.
In addition, iVAE assumes the modulationof class labels on latent distributions is limited within the exponential family distribution.
On thecontrary, our nonparametric conditions (Theorems 1,2,3) allow any kind of modulation caused byfixed, changing transition dynamics or observation changes without those strong assumptions on thedistribution of latent variables or noise distribution (i.e., SlowVAE [12]).
[Equation]
GCL [8] The formulation of the underlying sources in GCL is in Eq 30, which is also described by
Eqs.
4,15 in the original paper [8]:
[Equation]
where the latent processes are free from the exponential family distribution assumptions but stillconstrained within mutually-independent processes.
On the contrary, our work considers causallyrelated latent space in which cross causal relations between latent variables can be recovered.
Additionally, we want to mention that we provide an antenna tube in the schema of the proof in
Theorems 1-2-3, which is a more direct way of using sufficient variability conditions than [8].
LEAP [14] LEAP in Eq.
31 considers one special case of nonstationarity caused by changes innoise distributions, while our work can allow changing causal relations over context.
Furthermore,because LEAP assumes all latent processes are changed across contexts, it doesn’t use or benefitfrom the fixed time-delayed causal relations for identifiability.
On the contrary, our work exploits themodular distribution changes from the fixed causal dynamics, changing dynamics, and observationchanges, and hence our identifiability conditions are generally weaker than [14].
[Equation]
[Equation]
S1.4 Discussion of the Assumptions
We first explain and justify each critical assumption in the proposed conditions.
We then discuss howrestrictive or mild the conditions are in real applications.
S1.4.1 Linear Independence Condition
Our proposed linear independence condition is a combination of stationary identifiability conditions(Eq.
4) in each context u r , plus the identifiability conditions for nonrecurrent influences (Eq.
Thecondition is essential to make each row of the Jacobian matrix H t of the indeterminacy function ofthe learned latent space in Eq.
17 to have only one non-zero entry, thus making the learned latentvariables identifiable up to permutation and component-wise invertible transformations.
In stationary environments , this condition essentially states that, during the generation of z t , ifeither of the two conditions is satisfied, then the linear independence condition holds in general.(1)
If the history information z Hx = { z t − τ } Lτ =
up to maximum time lag L , and the process noise ϵ tare coupled in a nontrivial way for generating z t (e.g., heterogeneous noise process in Eq. 32)
, suchthat z Hx can modulate the variance or higher-order statistics of the conditional distribution p ( z t | z Hx ) ,then the linear independence condition generally holds;
[Equation]
(32) (2) If the latent transition is an additive noise model (then (1) is violated) but the process noise isnon-Gaussian, it will be extremely hard for the linear independence condition to be violated.
Roughlyspeaking, for a randomly chosen conditional density function p ( z k,t | z
t − 1 ) in which z k,t is notindependent from z t − 1 (i.e., there is temporal dependence in the latent processes) and which does notfollow an additive noise model with Gaussian noise, the chance for its specific second- and third-orderpartial derivatives to be linearly dependent is slim.
In nonstationary environments , this condition was introduced in GCL [8], namely, “sufficientvariability”, to extend the modulated exponential families [6] to general modulated distributions.
Essentially, the condition says that the nonstationary regimes u must have a sufficiently complexand diverse effect on the transition distributions.
In other words, if the underlying distributions arecomposed of relatively many domains of data, the condition generally holds true.
For instance, inthe linear Auto-Regressive (AR) model with Gaussian innovations where only the noise variance changes, the condition reduces to the statement in [38] that the variance of each noise term fluctuatessomewhat independently of each other in different nonstationary regimes.
Then the condition is easilyattained if the variance vector of noise terms in any regime is not a linear combination of variancevectors of noise terms in other regimes.
We further illustrate the condition using the example of modulated conditional exponential familiesin [8].
Let the log-pdf q ( z t |{ z t − τ } , u ) be a conditional exponential family distribution of order kgiven nonstationary regime u and history z
Hx = { z t − τ } :
[Equation]
where q i is the base measure, q ij is the function of the sufficient statistic, λ ij is the natural parameter,and log Z is the log-partition.
Loosely speaking, the sufficient variability holds if the modulation ofby u on the conditional distribution q ( z it | z Hx , u ) is not too simple in the following sense:
Higher order of k ( k > 1 ) is required.
If k = 1 , the sufficient variability cannot hold;2.
The modulation impacts λ ij by u must be linearly independent across regimes u .
The
sufficient statistics functions q ij cannot be all linear
i
e.
we
require higher
-order statistics.
Further details of this example can be found in Appendix B of [8].
In summary, we need themodulation by u to have diverse (i.
e
., distinct influences and complex impacts on the underlying
data generation process
Applicability By combining the stationary and nonstationary conditions, our proposed identifiability condition is generally mild, in the sense that if there is at least one regime r out of the m contextswhich satisfies the stationary identifiability conditions, OR, if the overall nonstationary influences arediverse and complex, thus satisfying the nonstationary identifiability conditions, the latent temporalcausal processes are identifiable.
For stationary conditions, the only situation where we find the latentprocesses unidentifiable is when the latent temporal transition is described by a Gaussian additivenoise model, which violates both (1) and (2).
However, for real-world data, it is very unlikely forthe process noise to be perfectly Gaussian.
Nonstationarity seems to be prominent in many kindsof temporal data.
For example, nonstationary variances are seen in EEG/MEG, and natural video,and are closely related to changes in volatility in financial time series [6].
The data that most likelysatisfy the nonstationary condition is a collection of multiple trials/segments of data with differenttemporal dynamics in between.
S1.4.2 Independent Noise Condition
The IN condition was introduced in the Structural Equation Model (SEM), which represents effect Yas a function of direct causes X and noise E :
[Equation] [Equation]
If X and Y do not have a common cause, as seen from the causal sufficiency assumption of structuralequation models in Chapter 1.4.1 of Pearl’s book [39], the IN condition states that the unexplainednoise variable E is statistically independent of cause X .
IN is a direct result of assuming causalsufficiency in SEM.
The main idea for the proof is that if IN is violated, then by the common causeprinciple [40], there exist hidden confounders that cause their dependence, thus violating the causalsufficiency assumption.
Furthermore, the noise terms in different variables are mutually independentfor a causally sufficient system with acyclic causal relations.
The main idea is that when the noiseterms are dependent, it is customary to encode such dependencies by augmenting the graph withhidden confounder variables [39], which means that the system is not causally sufficient.
This paper assumes that the underlying latent processes form a casually-sufficient system withoutlatent causal confounders.
Then, the process noise terms
ϵ
it are mutually independent, and moreover,the process noise terms
ϵ
it are independent of direct cause/parent nodes Pa ( z it ) because of timeinformation (the causal graph is acyclic because of the temporal precedence constraint).
Applicability Loosely speaking, if there are no latent causal confounders in the (latent) causalprocesses and the sampling frequency is high enough to observe the underlying dynamics, thenthe IN condition assumed in this paper is satisfied in a causally-sufficient system and, moreover,there is no instantaneous causal influence (because of the high enough resolution).
At the sametime, we acknowledge that there exist situations where the resolution is low and there appears tobe instantaneous dependence.
However, several pieces of work deal with causal discovery frommeasured time series in such situations; see.
e.g., [41, 42, 43, 44].
In case there are instantaneouscausal relations among latent causal processes, one would need additional sparsity or minimalityconditions to recover the latent processes and their relations, as demonstrated in [45, 46].
How toaddress the issue of instantaneous dependency or instantaneous causal relations in the latent processeswill be one line of our future work.
S1.4.3 Causal Influences between Observed Variables
Although causal discovery between observed variables is not the main focus of our work, our modelcan discover causal relations between observed variables as a special case, thanks to the nonlinearmixing function assumed in this paper.
In our formulation in Eq.
1, we assume the observations x tare nonlinear, invertible mixtures of latent processes z t .
However, if in the data generating process,one observed variable x i has direct causal edges with the other observed variable x j , the mixingfunction that generates x i and x j will just be reduced to identity mappings of z i and z j , which is aspecial case of the nonlinear invertible mixing function in Eq.
S1.5 Extension to Multiple Time Lags
For the sake of simplicity, we consider one lag for the latent processes in Section 3 (and onlyin Section 3).
Our identifiability proof can actually be applied for arbitrary lags directly.
Forinstance, in the stationary case in Eq. 4, one can simply re-define η kt ≜ log p ( z k,t | z Hx ) , wherez Hx denotes the lagged latent variables up to maximum time lag L .
We plug it into Eq. 4, and takederivatives with regard to z 1 ,t − τ , ..., z n,t − τ , which can be any latent temporal variables at lag τ ,instead of z 1 ,t − 1 , ..., z n,t − 1 .
If there exists one τ (out of the L lags) that satisfies the condition,then the stationary latent processes are identifiable.
Similarly, for Eq. 8, one can simply re-defineη _ kt ( u r ) ≜ log p ( z k,t | z Hx , u r ) and plug it into Eq. 8.
No extra changes are needed.
S
Experiment SettingsS2.1 DatasetsS2.1.1 Synthetic Dataset Generation
We consider three representative simulation settings to validate the identifiability results under fixedcausal dynamics, changing causal dynamics, and modular distribution shift which contains fixeddynamics, changing dynamics, and global changes together in the latent processes.
For syntheticdatasets with fixed and changing causal dynamics, we set latent size n = 8 .
For the modular shiftdataset, we add one dimension for global observation changes.
The lag number of the process is setto L = 2 .
The mixing function g is a random three-layer MLP with LeakyReLU units.
Fixed Causal Dynamics For the fixed causal dynamics.
We generate 100,000 data points accordingto Eq.
(6), where the latent size is n = 8 , lag number of the process is L = 2 .
We apply a 2-layer
MLP with LeakyReLU as the state transition function.
The process noise are sampled from i.i.d.
Gaussian distribution ( σ = 0 . 1 ).
The process noise terms are coupled with the history informationthrough multiplication with the average value of all the time-lagged latent variables.
Changing Causal Dynamics We use a Gaussian additive noise model with changes in the influencing strength as the latent processes.
To add changes, we vary the values of the first layer of the
MLP across the 20 segments and generate 7,500 samples for each segment.
The entries of the kernelmatrix of the first layer are uniformly distributed between [ − 1 , 1] in each domain.
Modular Distribution Shifts
The latent space of this dataset is partitioned into 6 fixed dynamicscomponents under the heterogeneous noise model, 2 changing components with changing causal dynamics and 1 component modulated by domain index only.
The fixed and changing dynamicscomponents follow the same generating procedures above.
The global change component is sampledfrom i.i.d Gaussian distribution whose mean and variance are modulated by domain index.
Inparticular, distribution mean terms are uniformly sampled between [ − 1 , 1] and variance terms areuniformly sampled between [0 . 01 , 1] .
S2.1.2 Real-world Dataset
Modified Cartpole
The Cartpole problem [30] “consists of a cart and a vertical pendulum attachedto the cart using a passive pivot joint.
The cart can move left or right.
The task is to prevent thevertical pendulum from falling by putting a force on the cart to move it left or right.
The action spaceconsists of two actions: moving left or right.”
Left
[Figure]
The original dataset [30] introduces “two change factors respectively for the state transition dynamicsθ dyn k : varying gravity and varying mass of the cart, and a change factor in the observation functionθ obs
k
that is the image noise level.
Fig.
S1 gives a visual example of Cartpole game, and the imagewith Gaussian noise.
The images of the varying gravity and mass look exactly like the original image.
Specifically, in the gravity case, we consider source domains with gravity g
= { 5 , 10 , 20 , 30 , 40 } .
We take into account both interpolation (where the gravity in the target domain is in the supportof that in source domains) with g = { 15 } , and extrapolation (where it is out of the support w.r.t.the source domains) with g = { 55 } .
Similarly, we consider source domains where the mass ofthe cart is m = { 0 . 5 , 1 . 5 , 2 . 5 , 3 . 5 , 4 . 5 } , while in target domains it is m = { 1 . 0 , 5 . 5 } .
In termsof changes on the observation function, we add Gaussian noise on the images with variance σ ={ 0 . 25 , 0 .
in source domains, and σ = { 0 . 5 , 2 . 75 } in target domains.
The detailedsettings in both source and target domains are in Table S1.”
[Table]
CMU-Mocap CMU MoCap ( http://mocap.cs.cmu.edu/ ) is an open-source human motioncapture dataset with various motion capture recordings (e.g., walk, jump, basketball, etc.)
performedby over 140 subjects.
In this work, we fit our model on 11 trials of “walk” recordings (Subject #8).
Skeleton-based measurements have 62 observed variables corresponding to the locations of joints(e.g., head, foot, shoulder, wrist, throat, etc.) of the human body at each time step.
S2.2 Mean Correlation Coefficient
MCC is a standard metric for evaluating the recovery of latent factors in ICA literature.
MCC firstcalculates the absolute values of the correlation coefficient between every ground-truth factor againstevery estimated latent variable.
Pearson correlation coefficients or Spearman’s rank correlationcoefficients can be used depending on whether componentwise invertible nonlinearities exist in therecovered factors.
The possible permutation is adjusted by solving a linear sum assignment problemin polynomial time on the computed correlation matrix.
S Implementation DetailsS3.1 Modular Prior Likelihood Derivation
Let us start with an illustrative example of stationary latent causal processes consisting of two timedelayed latent variables, i.e., z t =
[ z 1 ,t , z 2 ,t ] with maximum time lag L = 1 , i.e., z i,t = f i ( z t − 1 , ϵ i,t )with mutually independent noises.
Let us write this latent process as a transformation map
f
(notethat we overload the notation f for transition functions and for the transformation map):
[Equation]
By applying the change of variables formula to the map f , we can evaluate the joint distribution ofthe latent variables p ( z 1 ,t − 1 , z 2 ,t − 1 , z 1 ,t , z 2 ,t ) as:
[Equation]
36) where J f is the Jacobian matrix of the map f , which is naturally a low-triangular matrix:
[Equation]
Given that this Jacobian is triangular, we can efficiently compute its determinant as (cid:81)
i ∂z i,t ∂ϵ
i,t .
Furthermore, because the noise terms are mutually independent, and hence ϵ i,t ⊥ ϵ j,t for j ̸
= i andϵ t ⊥ z t − 1 , we can write the RHS of Eq.
36 as:
[Equation]
ϵ
[Equation]
⊥ ϵ
[Equation]
Finally, by canceling out the marginals of the lagged latent variables p ( z 1 ,t − 1 , z 2 ,t − 1 ) on both sides,we can evaluate the transition prior likelihood as:
[Equation]
Now we generalize this example and derive the modular prior likelihood below.
Fixed Causal Dynamics Let { f − 1 s } s =1 , 2 , 3 ... be a set of learned inverse fixed dynamics transitionfunctions that take the estimated latent causal variables in the fixed dynamics subspace and laggedlatent variables, and output the noise terms, i.e., ˆ ϵ s,t = f − 1 s (cid:0) ˆ z fix s,t , { ˆ z t − τ } (cid:1) .
Design transformation
A → B with low-triangular Jacobian as follows:
[Equation]
mapped to (cid:2)
ˆ z t −
[Equation]
z
[Equation] [Equation]
, with J A
[Equation]
=
[Equation]
I nL ∗ diag
[Equation] [Equation]
Similar to Eq.
[Equation]
we can obtain the joint distribution of the estimated fixed dynamics subspace as:
[Equation]
Because
[Table]
[Equation]
Changing Causal Dynamics
The differences from fixed dynamics are that the learned inversechanging dynamics transition functions take additional learned change factors of the context as inputarguments to out the noise terms, i.
[Equation]
[Equation] [Equation]
dyn k
[Equation] [Equation]
[Equation]
Observation Changes The global observation changes are captured by the learned inversef − 1 o ,which takes the estimated latent subspace and the learned change factors for global observation θ obs k of context k , and output random noise, i.e, ˆ ϵ o,t = f − 1 o (cid:0) ˆ z obs o,t
[Equation]
u k (cid:1) =
f − 1 c (cid:16) ˆ z
[Equation]
c,t , θ obs k (cid:17) .
[Equation] [Equation]
S3.2 Comparisons with AdaRL [30]
In terms of the implementation, in our work, we enforce the independent noise or conditionalindependence condition explicitly in Eq.
9 (derived in Appendix S3.1) for the identifiability of thelatent processes.
But disentanglement is not the main goal of AdaRL so it used Mixture Density
Network (MDN) to approximate the transition prior.
Our framework is simpler than AdaRL in theinference module (we use only x t to infer z t ) and the loss function (we don’t have the predictionbranch or the sparsity loss), thanks to the nonlinear ICA formulation in Eq. 1.
Our identifiabilityconditions do not rely on the sparsity constraints in the underlying data generating process.
S3.3 Network Architecture
We summarize our network architecture below and describe it in detail in Table S2 and Table S3.
S3.4 Hyperparameter and TrainingHyperparameter
Selection The hyperparameters include β , which is the weight of KLD terms,as well as the latent size n and maximum time lag L .
We use the ELBO loss to select the best pairof β because low ELBO loss always leads to high MCC.
We always set a larger latent size than thetrue latent size.
This is critical in real-world datasets because restricting the latent size will hurt thereconstruction performances and over-parameterization makes the framework robust to assumptionviolations.
For the maximum time lag L , we set it by the rule of thumb.
For instance, we use L = 2for temporal datasets with a latent physics process (e.g, cartpole, cmu-mocap).
Training Details
The models were implemented in PyTorch 1.8.1.
The VAE network is trainedusing AdamW optimizer for a maximum of 50 epochs and early stops if the validation ELBO lossdoes not decrease for five epochs.
A learning rate of 0.002 and a mini-batch size of 64 are used.
Wehave used three random seeds in each experiment and reported the mean performance with standarddeviation averaged across random seeds.
[Table]
Training Stability We have used several tricks to improve training stability: (1) we use a slightlylarger latent size than the true latent size for real-world datasets in order to make sure the meaninglatent variables are among the recovered latent; (2) we use AdamW optimizer as a regularizer toprevent training from being interrupted by overflow or underflow of variance terms of VAE; (3) weuse a large learning rate (0.002) during training.
Computing Hardware We used a machine with the following CPU specifications: Intel(R)
Core(TM) i7-7700K CPU @ 4.20GHz; 8 CPUs, four physical cores per CPU, a total of 32 logical
CPU units.
The machine has two GeForce GTX 1080 Ti GPUs with 11GB GPU memory.
Reproducibility We’ve included the code for the framework and all experiments in the supplement.
We plan to release our code under the MIT License after the paper review period.
[Figure]
Additional Experiment Results
[Figure] [Table]