Temporally Disentangled Representation Learning
Abstract
Recently in the ﬁeld of unsupervised representation learning, strong identiﬁability results for disentanglement of causally-related latent variables have been established by exploiting certain side information, such as class labels, in addition to independence.
However, most existing work is constrained by functional form assumptions such as independent sources or further with linear transitions, and distribution assumptions such as stationary, exponential family distribution.
It is unknown whether the underlying latent variables and their causal relations are identiﬁable if they have arbitrary, nonparametric causal inﬂuences in between.
In this work, we establish the identiﬁability theories of nonparametric latent causal processes from their nonlinear mixtures under ﬁxed temporal causal inﬂuences and analyze how distribution changes can further beneﬁt the disentanglement.
We propose TDRL , a principled framework to recover time-delayed latent causal variables and identify their relations from measured sequential data under stationary environments and under different distribution shifts.
Speciﬁcally, the framework can factorize unknown distribution shifts into transition distribution changes under ﬁxed and time-varying latent causal relations, and under global changes in observation.
Through experiments, we show that time-delayed latent causal inﬂuences are reliably identiﬁed and that our approach considerably outperforms existing baselines that do not correctly exploit this modular representation of changes.
Our code is available at: https://github.com/weirayao/tdrl .
Introduction
Causal reasoning for time-series data is a fundamental task in numerous ﬁelds [1, 2, 3].
Most existing work focuses on estimating the temporal causal relations among observed variables.
However, in many real-world scenarios, the observed signals (e g., image pixels in videos) do not have direct causal edges but are generated by latent temporal processes or confounders that are causally related.
Inspired by these scenarios, this work aims to uncover causally-related latent processes and their relations from observed temporal variables.
Estimating latent causal structure from observations, which we assume are unknown (but invertible) nonlinear mixtures of the latent processes, is very challenging.
It has been found in [4, 5] that without exploiting an appropriate class of assumptions in estimation, the latent variables are not identiﬁable in the most general case.
As a result, one cannot make causal claims on the recovered relations in the latent space.
Recently, in the ﬁeld of unsupervised representation learning, strong identiﬁability results of the latent variables have been established [6, 7, 8, 9, 10] by using certain side information in nonlinear
Independent Component Analysis (ICA), such as class labels, in addition to independence.
For time-series data, history information is widely used as the side information for the identiﬁability of latent processes.
To establish identiﬁability, the existing approaches enforce different sets of functional and distributional form assumptions as constraints in estimation; for example, (1) PCL [7], GCL [8], HM-NLICA [11] and SlowVAE [12] assume mutually-independent sources in the data generating process.
However, this assumption may severely distort the identiﬁability if the
[Table]
Applicable
[Table]
latent variables have time-delayed causal relations in between (i.e., causally-related process); (2)
SlowVAE [12] and SNICA [13] assume linear relations, which may distort the identiﬁability results if the underlying transitions are nonlinear, and (3) SlowVAE
[12] assumes that the process noise is drawn from Laplacian distribution; i-VAE [9] assumes that the conditional transition distribution is part of the exponential family.
However, in real-world scenarios, one cannot choose a proper set of functional and distributional form assumptions without knowing in advance the parametric forms of the latent temporal processes.
Our ﬁrst step is hence to understand under what conditions the latent causally processes are identiﬁable if they have nonparametric transitions in between.
With the proposed condition, our approach allows recovery of latent temporal causally-related processes in stationary environments without knowing their parametric forms in advance.
Mass
[Table]
[Equation]
[Table]
[Equation] [Equation]
[Equation]
Mass
[Equation]
T
[Equation]
[Figure]
Index
Transitions
Time
Change factors for transition dynamics and observations
[Figure]
Latent Causal Process Estimation
[Figure]
Learning.
We exploit ﬁxed causal dynamics and distribution changes from changing causal inﬂuences and observation changes to identify the underlying causal processes.
ˆ z i,t denotes the estimated latent process.
ˆ ✓ dyn r is the change factor for transition dynamics , i.e., representing mass and gravity in this example.
ˆ ✓ obs r is the change factor for observation , i.e., noise scale.
On the other hand, nonstationarity has greatly improved the identiﬁability results for learning the latent causal structure [14, 15, 16].
For instance, LEAP [14] established the identiﬁability of latent temporal processes, but in limited nonstationary cases, under the condition that the distribution of the noise terms of the latent processes varies across all segments.
Our second step is to analyze how distribution shifts beneﬁt our stationary condition and to extend our condition to a general nonstationary case.
Accordingly, our approach enables the recovery of latent temporal causal processes in a general nonstationary environment with time-varying relations such as changes in the inﬂuencing strength or switching some edges off [17] over time or domains.
Given the identiﬁability results, we propose a learning framework, called TDRL , to recover nonparametric time-delayed latent causal variables and identify their relations from measured temporal data under stationary environments and under nonstationary environments in which it is unknown in advance how the joint distribution changes across domains (we deﬁne it as “unknown distribution shifts”).
For instance, Fig. 1 shows an example of multiple video domains of a physical system under different mass, gravity, and environment rendering settings 1 .
With TDRL , the differences across segments are characterized by the learned change factors ˆ ✓ dyn r of domain r (note that domain index is given to the model) that encode changes in transition dynamics, and changes in observation or styles modeled by ˆ ✓ obs r (we use “causal dynamics” and “latent causal relations/inﬂuences” interchangeably).
We then present a generalized time-series data generative model that takes these change factors as arguments for modeling the distribution changes.
Speciﬁcally, we factorize unknown distribution shifts into transition distribution changes in stationary processes, time-varying latent causal relations, and global changes in observation by constructing partitioned latent subspaces, and propose provable conditions under which nonparametric latent causal processes can be identiﬁed from their nonlinear invertible mixtures.
We demonstrate through a number of realworld datasets, including video and motion capture data that time-delayed latent causal inﬂuences are reliably identiﬁed from observed variables under stationary environments and unknown distribution shifts.
Through experiments, we show that our approach considerably outperforms existing baselines that do not correctly leverage this modular representation of changes.
Our results demonstrate that causal representation learning from sequential data seems promising both in stationary and general nonstationary settings, in which the latent processes can have arbitrary, nonparametric causal inﬂuences in between.
Related Work
Causal Discovery from Time Series Inferring the causal structure from time-series data is critical to many ﬁelds including machine learning [1], econometrics [2], and neuroscience [3].
Most existing work focuses on estimating the temporal causal relations between observed variables.
For this task, constraint-based methods [18] apply the conditional independence tests to recover the causal structures, while score-based methods [19, 20] deﬁne score functions to guide a search process.
Furthermore, [21, 22] propose to fuse both conditional independence tests and score-based methods.
The Granger causality [23] and its nonlinear variations [24, 25] are also widely used.
Nonlinear ICA for Time Series Temporal structure and nonstationarities were recently used to achieve identiﬁability in nonlinear ICA.
Time-contrastive learning (TCL [6]) used the independent sources assumption and leveraged sufﬁcient variability in variance terms of different data segments.
Permutation-based contrastive (PCL [7]) proposed a learning framework which discriminates between true independent sources and permuted ones, and identiﬁable under the uniformly dependent assumption.
HM-NLICA [11] combined nonlinear ICA with a Hidden Markov Model (HMM) to automatically model nonstationarity without manual data segmentation.
i-VAE [9] introduced VAEs to approximate the true joint distribution over observed and auxiliary nonstationary regimes.
Their work assumes that the conditional distribution is within exponential families to achieve the identiﬁability of the latent space.
The most recent literature on nonlinear ICA for time-series includes LEAP [14] and (i-)CITRIS [26, 27].
LEAP proposed a nonparametric condition leveraging the nonstationary noise terms.
However, all latent processes are changed across contexts and the distribution changes need to be modeled by nonstationary noise and it does not exploit the stationary nonparametric components for identiﬁability.
Alternatively, CITRIS proposed to use intervention target information for identiﬁcation of scalar and multidimensional latent causal factors.
This approach does not suffer from functional or distributional form constraints, but needs access to active intervention.
Problem Formulation 3.1 Time Series Generative Model
Stationary Model As a fundamental case, we ﬁrst present a regular, stationary time-series generative process where the observations x t comes from a nonlinear (but invertible) mixing function g that maps the time-delayed causally-related latent variables z t to x t .
The latent variables or processes z t have stationary, nonparametric time-delayed causal relations.
Let ⌧ be the time lag:
[Equation]
Stationary noise
[Equation]
Note that with nonparametric causal transitions, the noise term ✏ it ⇠ p ✏ i (where p ✏ i denotes the distribution of ✏ it ) and the time-delayed parents Pa ( z it ) of z
it (i.e., the set of latent factors that directly cause z it ) are interacted and transformed in an arbitrarily nonlinear way to generate z it .
Under stationarity assumptions, the mixing function g , the transition functions
f i and the noise distributions p ✏ i are invariant.
Finally, we assume that the noise terms are mutually-independent (i.e., spatially and temporally independent), which implies that instantaneous causal inﬂuence between latent causal processes is not allowed by the formulation.
The stationary time-series model in the fundamental case is used to establish the identiﬁability results under ﬁxed causal dynamics in Section 4.1.
Nonstationary Model We further consider two violations of the stationarity assumptions in the fundamental case, which lead to two nonstationary time series models.
Let u denote the domain or regime index.
Suppose there exist m regimes of data, i.e., u r with r = 1 , 2 , ..., m , with unknown distribution shifts.
In practice, the changing parameters of the joint distribution across domains often lie in a low-dimensional manifold [28].
Moreover, if the distribution is causally factorized, the distributions are often changed in a minimal and sparse way [29].
Based on these assumptions, we introduce the low-dimensional minimal change factor ( ✓ dyn r , ✓ obs r ) , which was proposed in [30], to respectively capture distribution shifts in transition functions and observation.
The vector ✓ r = ( ✓ dyn r , ✓ obs r ) has a constant value in each domain but varies across domains.
The formulation of the nonstationary time-series model is in line with [30].
The nonstationary model is used to establish the identiﬁability results under nonstationary cases in Section 4.2, where we show that the violation of stationarity in both ways can even further improve the identiﬁability results.
We ﬁrst present the two nonstationary cases.
(1) Changing Causal Dynamics .
The causal inﬂuences between the latent temporal processes are changed across domains in this setting.
We model it by adding the transition change factors ✓ dyn r as inputs to the transition function: z it =
[Equation]
i
[Equation]
⌘ .
(2) Observation Changes.
The global properties of the time series (e.g., video styles) are changed across domains in this setting.
Our model captures them using latent variables that represent global styles; these latent variables are generated by a bijection f i that transforms the noise terms ✏ i,t into the latent with change factor ✓ obs r : z i,t = f i ⇣ ✓ obs r , ✏ i,t ⌘ .
Finally, we can deal with a more general nonstationary case by combining the three types of latent processes in the latent space in a modular way.
Modular Distribution Shifts.
[Equation]
The latent space has three blocks z t =
( z ﬁx t , z chg t , z obs t ) where z ﬁx s,t denotes the s th component of the ﬁxed dynamics parts, z chg c,t denotes the c th component of the changing dynamics parts, and z obs o,t is the o th component of the global changes.
The functions [ f s , f c , f o ] capture ﬁxed and changing transitions and observation changes for each dimension of z t in Eq.
3.2 Identiﬁability of Latent Causal Processes and Time-Delayed Latent Causal Relations
We deﬁne the identiﬁability of time-delayed latent causal processes in the representation function space in Deﬁnition 1 .
Furthermore, if the estimated latent processes can be identiﬁed at least up to permutation and component-wise invertible nonlinearities, the latent causal relations are also immediately identiﬁable because conditional independence relations fully characterize time-delayed causal relations in a time-delayed causally sufﬁcient system, in which there are no latent causal confounders in the (latent) causal processes.
Note that invertible component-wise transformations on latent causal processes do not change their conditional independence relations.
Deﬁnition 1 (Identiﬁable Latent Causal Processes) .
Formally let { x t } T t =1 be a sequence of observed variables generated by the true temporally causal latent processes speciﬁed by ( f i , ✓ r , p ( ✏ i ) , g ) given in Eq.
A learned generative model ( ˆ f i , ˆ ✓ r , ˆ p ( ✏ i ) , ˆ g ) is observationally equivalent to ( f i , ✓ r , p ( ✏ i ) , g ) if the model distribution p ˆ f, ˆ ✓ r , ˆ p ✏ , ˆ g ( {
x t } Tt =1 ) matches the data distribution p f, ✓ r ,p ✏ , g ( {
x t } Tt =1 ) everywhere.
We say latent causal processes are identiﬁable if observational equivalence can lead to identiﬁability of the latent variables up to permutation ⇡ and component-wise invertible transformation T :
[Equation]
where X is the observation space.
Identiﬁability Theory
We establish the identiﬁability theory of nonparametric time-delayed latent causal processes under three different types of distribution shifts.
W.l.o.g.
, we consider the latent processes with maximum time lag L = 1 .
The extentions to arbitrary time lags are discussed in Appendix S1.5.
Let k be the element index of the latent space z t and the latent size is n .
In particular, (1) under ﬁxed temporal causal inﬂuences, we leverage the distribution changes p ( z k,t | z t  1 ) for different values of z t  1 ; (2) when the underlying causal relations change over time, we exploit the changing causal inﬂuences on p ( z k,t | z t  1 , u r ) under different domain u r , and (3) under global observation changes, the nonstationarity p ( z k,t | u r ) under different values of u r is exploited.
The proofs are provided in
Appendix S1.
The comparisons between existing theories are in Appendix S1.3.
4.1 Identiﬁability under Fixed Temporal Causal Inﬂuence
Let ⌘ kt , log p ( z k,t | z t  1 ) .
Assume that ⌘ kt is twice differentiable in z k,t and is differentiable in z l,t  1 , l = 1 , 2 , ..., n .
Note that the parents of z k,t may be only a subset of z t  1 ; if z l,t  1 is not a parent of z k,t , then @⌘ kt @
z l,t  1 = 0 .
Below we provide a sufﬁcient condition for the identiﬁability of z t , followed by a discussion of speciﬁc unidentiﬁable and identiﬁable cases to illustrate how general it is.
Theorem 1 (Identiﬁablity under a Fixed Temporal Causal Model) .
Suppose there exists invertible function ˆ g that maps x t to ˆ z t , i.e.,
[Equation]
such that the components of ˆ z t are mutually independent conditional on ˆ z t  1 .
Let v k,t ,
[Equation]
z 2 k,t @ z 1 ,t  1 , @ 3 ⌘ kt @
z 2 k,t @ z 2 ,t  1 , ..., @ 3 ⌘ kt @
z 2 k,t @
z n,t ⌘ | .
If for each value of z t , v 1 ,t , ˚ v 1 ,t , v 2 ,t , ˚ v 2 ,t , ..., v n,t , ˚ v n,t , as 2 n vector functions in z 1 ,t  1 , z 2 ,t  1 , ..., z n,t  1 , are linearly independent, then z t must be an invertible, component-wise transformation of a permuted version of ˆ z t .
The linear independence condition in Theorem 1 is the core condition to guarantee the identiﬁability of z t from the observed x t .
To make this condition more intuitive, below we consider speciﬁc unidentiﬁable cases, in which there is no temporal dependence in z t or the noise terms in z t are additive Gaussian, and two identiﬁable cases, in which z t has additive, heterogeneous noise or follows some linear, non-Gaussian temporal process.
Let us start with two unidentiﬁable cases.
In case N 1 , t t is an independent and identically distributed (i.i.d.) process, i.e., there is no causal inﬂuence from any component of z t  1 to any z k,t .
In this case, v k,t and ˚ v k,t (deﬁned in Eq. 4) are always 0 for k = 1 , 2 , ..., n , since p ( z k,t
| z t  1 ) does not involve z t  1 .
So the linear independence condition is violated.
In fact, this is the regular nonlinear
ICA problem with i.i.d.
data, and it is well-known that the underlying independent variables are not identiﬁable [5].
In case N 2 , all z k,t follow an additive noise model with Gaussian noise terms, i.e.,
[Equation]
5) where q is a transformation and the components of the Gaussian vector ✏ t are independent and also independent from z t  1 .
Then @ 2 ⌘ kt @
z 2 k,t is constant, and
[Equation]
0 , violating the linear independence condition.
In the following proposition we give some alternative solutions and verify the unidentiﬁability in this case.
Proposition 1 (Unidentiﬁability under Gaussian Noise) .
Suppose x t = g
( z t ) was generated by
Eq.
5, where the components of ✏ t are mutually independent Gaussian and also independent from z t  1 .
Then any ˆ z t =
D 1 UD 2 · z t , where D 1 is an arbitrary non-singular diagonal matrix, U is an arbitrary orthogonal matrix, and D 2 is a diagonal matrix with V ar  1 / 2 ( ✏ k,t ) as its k th diagonal entry, is a valid solution to satisfy the condition that the components of ˆ z t are mutually independent conditional on ˆ z t  1 .
Roughly speaking, for a randomly chosen conditional density function p ( z k,t | z t  1 ) in which z k,t is not independent from z t  1 (i.e., there is temporal dependence in the latent processes) and which does not follow an additive noise model with Gaussian noise, the chance for its speciﬁc second- and third-order partial derivatives to be linearly dependent is slim.
Now let us consider two cases in which the latent temporally processes z t are naturally identiﬁable.
First, consider case Y 1 , where z k,t follows a heterogeneous noise process, in which the noise variance depends on its parents:
[Equation]
Here we assume ✏ k,t is standard Gaussian and ✏ 1 ,t , ✏ 2 ,t , .., ✏ n,t are mutually independent and independent from z t  1 .
1 b k , which depends on z t  1 , is the standard deviation of the noise in z k,t .
(For conciseness, we drop the argument of b k and q k when there is no confusion.)
Note that in this model, if q k is 0 for all k = 1 , 2 , ..., n , it reduces to a multiplicative noise model.
The identiﬁability result of z t is established in the following corollary.
Corollary 1 (Identiﬁability under Heterogeneous Noise) .
Suppose x t = g
( z t ) was generated according to Eq. 6, and Eq.
3 holds true.
If b k · @ b k @
z t  1 and b k · @ b k @
z t  1 ( z k,t  q k )
b 2 k · @ q k @
z t  1 , with k = 1 , 2 , ..., n , which are in total 2 n function vectors in z t  1 , are linearly independent, then z t must be an invertible, component-wise transformation of a permuted version of ˆ z t .
Let us then consider another special case, denoted by Y 2 , with a linear, non-Gaussian temporal model for z t : the latent processes follow Eq. 5, with q being a linear transformation and ✏ k,t following a particular class of non-Gaussian distributions.
The following corollary shows that z t is identiﬁable as long as each z k,t receives causal inﬂuences from some components of z t  1 .
Corollary 2 (Identiﬁability under a Speciﬁc Linear, Non-Gaussian Model for Latent Processes) .
Suppose x t = g
( z t ) was generated according to Eq.
5, in which q is a linear transformation and for each z k,t , there exists at least one k 0 such that c k,k 0 , @
z k,t @
z k 0 ,t  1 6 = 0 .
Assume the noise term ✏ k,
[Equation]
follows a zero-mean generalized normal distribution:
[Equation]
If Eq. 3 holds, then z t must be an invertible, component-wise transformation of permuted ˆ z t .
4.2 Further Beneﬁts from Changing Causal Inﬂuences
Let ⌘ kt ( u r ) , log p ( z k,t | z t  1 , u r ) where r = 1 , ..., m .
LEAP [14] established the identiﬁability of the latent temporal causal processes z t in certain nonstationary cases, under the condition that the noise term in each z k,t , relative to its parents in z t  1 , changes across m contexts corresponding to u = u 1 , u 2 , ..., u m .
Here we show that the identiﬁability result shown in the previous section can further beneﬁt from nonstationarity of the causal model, and that our identiﬁability condition is generally much weaker than that in [14]: we allow changes in the noise term or causal inﬂuence on z k,t from its parents in z t  1 , and our “sufﬁcient variability" condition is just a necessary condition for that in [14] because of the additional information that one can leverage.
Let v k,t ( u r ) be v k,t , which is deﬁned in Eq.
4, in the u r context.
Similarly, Let ˚ v k,t ( u r ) be ˚ v k,t in the u r context.
Let s k,t ,
[Equation]
where  2 r = @ 2 ⌘ kt ( u r ) @
z 2 k,t
[Equation]
u r  1 ) @
z 2 k,t and  r = @⌘ kt ( u r ) @
z k,t  @⌘ kt ( u r  1 ) @
z k,t .
As provided below, in our case, the identiﬁablity of z t is guaranteed by the linear independence of the whole function vectors s k,t and ˚ s k,t , with k = 1 , 2 , ..., n .
However, the identiﬁability result in [14] relies on the linear independence of only the last m  1 components of s k,t and ˚ s k,t with k = 1 , 2 , ..., n ; this linear independence is generally a much stronger and restricted condition.
Theorem 2 (Identiﬁability under Changing Causal Dynamics) .
Suppose x t = g
( z t ) and that the conditional distribution p ( z k,t | z t  1 ) may change across m values of the context variable u , denoted by u 1 , u 2 , ..., u m .
Suppose the components of z t are mutually independent conditional on z t  1 in each context.
Assume that the components of ˆ z t produced by Eq.
3 are also mutually independent conditional on ˆ z t  1 .
If the 2 n function vectors s k,t and ˚ s k,t , with k = 1 , 2 , ..., n , are linearly independent, then ˆ z t is a permuted invertible component-wise transformation of z t .
Theorem 3 (Identiﬁability under Observation Changes) .
Suppose x t = g
( z t ) and that the conditional distribution p ( z k,t | u ) may change across m values of the context variable u , denoted by u 1 , u 2 , ..., u m .
Suppose the components of z t are mutually independent conditional on u in each context.
Assume that the components of ˆ z t produced by Eq.
3 are also mutually independent conditional on ˆ z t  1 .
If the 2 n function vectors s k,t and ˚ s k,t , with k = 1 , 2 , ..., n , are linearly independent, then ˆ z t is a permuted invertible component-wise transformation of z t .
Corollary 3 (Identiﬁability under Modular Distribution Shifts) .
Assume the data generating process in Eq.
If the three partitioned latent components z t = ( z ﬁx t , z chg t , z obs t ) respectively satisfy the conditions in Theorem 1, Theorem 2, and Theorem 3, then z t must be an invertible, component-wise transformation of a permuted version of ˆ z t .
Our Approach 5.1 TDRL : Temporally Disentangled Representation Learning
Given our identiﬁability results, we propose TDRL framework to estimate the latent causal dynamics under modular distribution shifts, by extending Sequential Variational Auto-Encoders [31] with tailored modules to model different distribution shifts, and enforcing the conditions in Sec. 4 as constraints.
We give the estimation procedure of the latent causal dynamics model in Eq. 1.
The model architecture is showcased in Fig.
The framework has the following three major components.
The implementation details are in Appendix S3.3.
Speciﬁcally, we leverage the partitioned estimated latent subspaces ˆ z t =
(ˆ z ﬁx t , ˆ z chg t , ˆ z obs t ) and model their distribution changes in conditional transition priors.
We use [ f s , f c , f o ] to capture causal relations as in Eq. 1, where f s captures ﬁxed causal inﬂuences, f c for changing causal inﬂuences, and f o for observation changes.
Accordingly, we learn [ ˆ f  1 s , ˆ f  1 c , ˆ f  1 o ] to output random process noise from the estimated direct cause (lagged states ˆ z Hx ) and effect (current states) variables ˆ z t .
[Figure]
Change Factor Representation
We learn to embed domain index u r into low-dimensional change factors (ˆ ✓ dyn r , ˆ ✓ obs r ) in Fig.
2C and insert them as external inputs to the (inverse) dynamics function ˆ f  1 c (ˆ ✓ dyn r ) , or the observation bijector ˆ f  1 o (ˆ ✓ obs r ) , respectively.
And hence the distribution shifts are captured and utilized in the implementation.
Modular Prior Network We follow standard conditional normalizing ﬂow formulation [32, 14].
Let z Hx denote the lagged latent variables up to maximum time lag L .
In particular, for 1) ﬁxed causal dynamics processes ˆ z ﬁx t , their transition priors are obtained by ﬁrst learning inverse transition functions
f  1 s that take the estimated latent variables and output random noise terms, and applying the change of variables formula to the transformation: p
[Equation]
; for 2) changing causal dynamics , we evaluate p
[Equation]
by learning a holistic inverse dynamics ˆ f  1 c that takes the estimated change factors for dynamics ˆ ✓ dyn r as inputs, and similarly for 3) observation changes ˆ z obs t , we learn to project them to invariant noise terms by ˆ f  1 o which takes the change factors ✓ obs r as arguments, and obtains p (ˆ z obs o,t | u r ) =
p ✏ o
[Equation]
as the prior.
Conditional independence of the estimated latent variables p (ˆ z t | ˆ z Hx ) is enforced by summing up all estimated component densities when obtaining the joint p ( z t | z Hx , u ) in Eq. 9.
Given that the Jacobian is lower-triangular, we can compute its determinant as the product of diagonal terms.
The detailed derivations are given in Appendix S3.1.
[Equation]
[Figure]
[Equation]
Factorized Inference
We infer the posteriors of each time step q (ˆ z t | x t ) using only the observation at that time step, because in Eq.
1, x t preserves all the information of the current system states so the joint probability q ( ˆz 1: T | x 1: T ) can be factorized into product of these terms.
We approximate the posterior q ( ˆz t | x t ) with an isotropic Gaussian with mean and variance from the inference network.
Optimization We train TDRL using the ELBO objective L ELBO = 1 N
P i 2 N L Recon   L KLD , in which we use mean-squared error (MSE) for the reconstruction likelihood L Recon .
The KL divergence
L KLD is estimated via a sampling approach since with a learned nonparametric modular transition prior, the distribution does not have an explicit form.
Speciﬁcally, we obtain the log-likelihood of the posterior, evaluate the prior log p (ˆ z t | ˆ z Hx , u r ) in Eq. 9, and compute their mean difference in the dataset as the KL loss: L KLD =
E ˆz t ⇠ q ⇣ ˆz ( i ) t | x ( i ) t ⌘ log q ( ˆz t | x t )  log p (ˆ z t | ˆ z
Hx , u r ) .
5.2 Causal Visualization
For visualization purposes, when the underlying latent processes have sparse causal relations, we ﬁt
LassoNet [33] on the latent processes recovered by TDRL to interpret the causal relations.
Speciﬁcally, we ﬁt LassoNet to predict ˆ z t using the estimated history information ˆ z Hx =
ˆ z t  ⌧ } L ⌧ =1 up to maximum time lag L .
Note that this postprocessing step is optional – the latent causal relations have already been captured in the learned transition functions in TDRL .
Also, our identiﬁability conditions do not rely on the sparsity of causal relations in the latent processes.
Experiments
We evaluate the identiﬁability results of TDRL on a number of simulated and real-world time-series datasets.
We ﬁrst introduce the evaluation metrics and baselines.
(1) Evaluation Metrics.
To evaluate the identiﬁability of the latent variables, we compute Mean Correlation Coefﬁcient (MCC) on the test dataset.
MCC is a standard metric in the ICA literature for continuous variables which measure the identiﬁability of the learned latent causal processes.
MCC is close to 1 when latent variables are identiﬁable up to permutation and component-wise invertible transformation in the noiseless case.
(2) Baselines.
Nonlinear ICA methods are used: (1) BetaVAE [34] which ignores both history and nonstationarity information; (2) iVAE [9] and TCL [6] which leverage nonstationarity to establish identiﬁability but assumes independent factors, and (3) SlowVAE
[12] and PCL [7] which exploit temporal constraints but assume independent sources and stationary processes, and (4) LEAP [14] which assumes nonstationary, causal processes but only models nonstationary noise.
Two other disentangled deep state-space models with nonlinear dynamics models: Kalman VAE (KVAE [35]) and Deep Variational Bayes Filters (DVBF [36]), are also used for comparisons.
[Table]
6.1 Simulated Experiments
We generate synthetic datasets that satisfy our identiﬁability conditions in the theorems following the procedures in Appendix S2.1.1.
As in Table 2, our framework can recover the latent processes under ﬁxed dynamics (heterogeneous noise model), under changing causal dynamics [Table] and under modular distribution shifts with high MCCs (>0.95).
The baselines that do not exploit history (i.e.,  VAE, i-VAE, TCL), with independent source assumptions (SlowVAE, PCL), considers limited nonstationary cases (LEAP) distort the identiﬁability results.
KVAE and DVBF achieve MCCs (0.8) under ﬁxed dynamics but distorts the identiﬁability under changing dynamics and modular shift setting because they don’t model changing causal relations and observation changes.
6.2 Real-world Applications
Video Data – Modiﬁed Cartpole Environment
We evaluate TDRL on the modiﬁed cartpole [30] video dataset and compare the performances with the baselines.
Modiﬁed Cartpole is a nonlinear dynamical system with cart positions x t and pole angles ✓ t as the true state variables.
The dataset descriptions are in Appendix S2.1.2.
We use 6 source domains with different gravity values g = { 5 , 10 , 15 , 20 , 25 , 30 } .
Together with the 2 discrete actions (i.e., left and right), we have 12 segments of data with changing causal dynamics.
We ﬁt TDRL with two-dimensional change factors ✓ dyn r .
We set the latent size n = 8 and the lag number L = 2 .
In Fig.
3, the latent causal processes are recovered, as seen from (a) high MCC for the latent causal processes; (b) the latent factors are estimated up to component-wise transformation; (c) TDRL outperforms the baselines and (d) the latent traversals conﬁrm the two recovered latent variables correspond to the position and pole angle.
[Figure]
Motion Capture Data – CMU-Mocap
We experimented with another real-world motion capture dataset (CMU-Mocap).
The dataset descriptions are in Appendix S2.1.2.
We ﬁt TDRL with 11 trials of motion capture data for subject #8.
The 11 trials contain walk cycles with very different styles (e.g., slow walk, stride).
We set latent size n = 3 and lag number L = 2 .
The differences between trials are captured through learning the 2-dimensional change factors for each trial.
In Fig. 4(a), the learned change factors group similar walk styles into clusters; in Panel (c), three latent variables (which seem to be pitch, yaw, roll rotations) are found to explain most of the variances of human walk cycles.
The learned latent coordinates (Panel b) show smooth cyclic patterns with differences among different walking styles.
For the discovered skeleton (Panel d), roll and pitch of walking are found to be causally-related while yaw has independent dynamics.
[Figure]
Conclusion
In this paper, without relying on parametric or distribution assumptions, we established identiﬁability theories for nonparametric latent causal processes from their observed nonlinear mixtures in stationary environments and under unknown distribution shifts.
The basic limitation of this work is that the underlying latent processes are assumed to have no instantaneous causal relations but only time-delayed inﬂuences.
If the time resolution of the observed time series is the same as the causal frequency (or even higher), this assumption naturally holds.
However, if the resolution of the time series is much lower, then it is usually violated and one has to ﬁnd a way to deal with instantaneous causal relations.
On the other hand, it is worth mentioning that the assumptions are generally testable; if the assumptions are actually violated, one can see that the results produced by our method may not be reliable.
Extending our theories and framework to address the issue of instantaneous dependency or instantaneous causal relations will be one line of our future work.
In addition, Empirically exploration of the merits of the identiﬁed latent causal graph in terms of few-shot transfer to new environments [37], domain adaptation [38], forecasting [39, 40] and control [30] is also one important future step.
Acknowledgment
Kun Zhang was partially supported by the National Institutes of Health (NIH) under Contract
R01HL159805, by the NSF-Convergence Accelerator Track-D award #2134901, by a grant from
Apple Inc., and by a grant from KDDI Research Inc.