Multimodal Teacher Forcing for Reconstructing Nonlinear Dynamical Systems
Abstract
Many, if not most, systems of interest in science are naturally described as nonlinear dynamical systems (DS).
Empirically, we commonly access these systems through time series mea- surements, where often we have time series from different types of data modalities simultaneously.
For instance, we may have event counts in addition to some continuous signal.
While by now there are many powerful machine learning (ML) tools for integrating different data modalities into predictive models, this has rarely been approached so far from the perspective of uncovering the underlying, data-generating DS (aka DS reconstruction).
Recently, sparse teacher forcing (TF) has been sug- gested as an efficient control-theoretic method for dealing with exploding loss gradients when training ML models on chaotic DS.
Here we incorporate this idea into a novel recurrent neural network (RNN) training framework for DS reconstruction based on multimodal variational autoencoders (MVAE).
The forcing signal for the RNN is generated by the MVAE which integrates different types of simultaneously given time series data into a joint latent code optimal for DS reconstruction.
We show that this training method achieves significantly better reconstructions on multimodal datasets generated from chaotic DS benchmarks than various alternative methods.
Introduction
For many temporally evolving phenomena in physics, biology, or the social sciences, we have only limited knowledge about the generating dynamical mechanisms.
Inferring these from data is a core interest in any scientific discipline.
It is also practically highly relevant for predicting important changes in system dynamics, like tipping points in climate systems (Bury et al. 2021; Patel and Ott 2022).
In recent years, a variety of methods for recovering dynamical systems (DS) directly and automatically from time series observations has been proposed (Brunton, Proctor, and Kutz 2016;
Vlachas et al.
; Koppe et al. 2019b), mostly based on recurrent neural networks (RNNs) for approximating the unknown governing equations of the true DS.
However, almost all of these methods assume that observed time series come as continuous signals with Gaussian noise, except for one recent study that also considered non-Gaussian, like categorical or sparse count data for DS reconstruction (Kramer et al.
Time series data from discrete random processes are in fact quite commonplace in many areas, e.g., in the medical domain (electronic health records, smartphone-based data) (Koppe et al. 2019a), neuroscience (behavioral responses) (Durstewitz, Koppe, and Thurm 2022), or climate science (event counts) (Peyre et al. 2020).
While it was shown that, in principle, integrating such different measurement modalities can help to improve DS reconstruction (Kramer et al. 2022), major issues remain.
Arguably one of the most challenging aspects is the ‘exploding and vanishing gradient problem (EVGP)’
(Bengio, Simard, and Frasconi 1994; Hochreiter and Schmidhuber 1997), a problem that cannot easily be avoided when attempting to capture natural systems with chaotic dynamics due to the exponential divergence of trajectories (Mikhaeil, Monfared, and Durstewitz 2022).
Thus, rather than trying to conquer the EVGP by RNN design, control-theoretic methods based on sparse teacher forcing (TF) were recently suggested to address the EVGP directly in BPTT-based training (Mikhaeil, Monfared, and
Durstewitz 2022).
The idea here is to guide the training process by pulling the reconstruction algorithm ‘back on track’ at times determined from the observed system’s maximal
Lyapunov exponent, i.e. by replacing the RNN’s latent states by states inferred from the current observations.
While this turned out to be a simple yet powerful remedy for reconstructing systems with chaotic dynamics, outperforming many other state-of-the-art algorithms (Brenner et al. 2022), it remained unclear how to harvest this idea for non-Gaussian observations as it relies on an invertible linear-Gaussian observation model.
Hence, one major contribution of the present work is to make the sparse-TF approach amenable to other than continuous Gaussian observations.
Another contribution is a novel formulation of the multimodal data integration problem for DS reconstruction: Building on the success of variational autoencoders (VAEs) for multimodal integration (Baltruˇsaitis, Ahuja, and Morency 2017; Wu and Goodman 2018; Sutter, Daunhawer, and Vogt 2021), we use multimodal VAEs (MVAEs) to construct a common latent representation from random variables following different distributional models.
However, rather than constructing a sequential VAE process that directly operates on this latent code, as in Kramer et al.
(2022), here the purpose of the MVAE is to provide a TF signal that acknowledges all recorded data modalities.
Thus, the MVAE learns a variational approximation to a latent distribution that yields the joint distribution over the observed multimodal data, which is then used as a TF signal in the latent space of a simultaneously trained RNN.
This makes the insights derived from sparse-TF applicable to a fully probabilistic
MVAE training framework, called MVAE-TF in here, which – unlike classical BPTT-TF – also explicitly allows for process (dynamical) noise and uncertainty estimates.
At the same time, it leaves the distributional assumptions of the individual observed modalities intact.
Related Work
Dynamical Systems Reconstruction In DS reconstruction the goal is to infer or learn a model of the nonlinear DS that produced a set of observed quantities such that invariant temporal and geometrical properties of the data-generating DS are preserved.
This is highly relevant for both science and engineering.
In general, nonlinear DS, especially when highdimensional, noisy, and chaotic, are challenging to learn and difficult to analyze (Champion et al. 2019).
RNNs are popular machine learning tools for modeling DS (Park et al. 2022), and various RNN architectures such as Reservoir Computing (RC)
(Pathak et al. 2018), Long-Short-Term-Memory networks (LSTMs) (Vlachas et al. 2018), or piecewise-linear
RNNs (PLRNNs) (Koppe et al. 2019b; Schmidt et al. 2021) have been employed for DS reconstruction.
However, with few exceptions (Kramer et al. 2022), almost all of the previously designed methods for DS reconstruction have focused solely on the case where all observations are continuous with
Gaussian noise.
RNN models trained with classical BPTT suffer from the EVGP (Bengio, Simard, and Frasconi 1994;
Hochreiter and Schmidhuber 1997; Pascanu, Mikolov, and
Bengio 2013).
Recent theoretical work (Mikhaeil, Monfared, and Durstewitz 2022) has shown that this problem is particularly severe when training on time series from chaotic systems (as almost always the case for complex physical or biological systems), as in this case loss gradients inevitably diverge.
While many specific RNN architectures have been proposed to deal with the EVGP (Kerg et al. 2019; Chang et al. 2019;
Kag, Zhang, and Saligrama 2020; Rusch and Mishra 2021), most of them are not suitable for DS reconstruction as their specific form or parameterization prevents many important
DS phenomena (like chaos) by design.
Instead, Mikhaeil,
Monfared, and Durstewitz (2022) suggested a variant of TF (Williams and Zipser 1989; Pearlmutter 1990), called sparse
TF, that strikes an optimal balance between learning relevant long time scales while avoiding too furiously diverging gradients.
At present, this method, however, relies on a continuous linear-Gaussian assumption for connecting the data to the latent process.
Generative Models for Multimodal Data Integration
Variational generative models are powerful methods for learning latent representations of joint distributions across many data types in an unsupervised fashion.
Variational autoencoders (VAE) (Kingma and Welling 2014; Rezende, Mohamed, and Wierstra 2014) are one popular variant which naturally lends itself to multimodal settings (Baltruˇsaitis,
Ahuja, and Morency 2017; Wu and Goodman 2018; Sutter,
Daunhawer, and Vogt 2021) and a sequential formulation (Bayer et al. 2021; Girin et al. 2021; Bai, Wang, and Gomes 2021).
Longitudinal autoencoders have been proposed (Ramchandran et al. 2021) to model temporal correlations in latent space with Gaussian process priors, and have also been extended to multimodal data ( ¨O˘gretir et al. 2022).
Other generative models for sequential data include state space models that have been applied for posterior inference of latent state paths z t ∼ p ( z t | x 1: T ) of DS given time series observations { x 1: T } (Ghahramani and Roweis 1998; Durstewitz 2017;
Pandarinath et al.
Zhao and Park 2020;
Kim et al. 2021).
Through their probabilistic formulation, they can account for uncertainty in the model formulation or latent process itself and yield the full distribution over latent states (Karl et al.
Many natural and engineered systems are observed through multiple channels with distinct statistical properties simultaneously.
For instance, in climate science we may have simultaneous records of temperatures and counts of extreme weather events like tornados.
Multimodal data integration can improve model inference and reveal interesting connections between observed modalities (Liang et al. 2015).
Multimodal models have also been developed for time series forecasting (Antelmi et al. 2018;
Bhagwat et al.
; Dezfouli et al.
; Shi et al.
2021; Sutter, Daunhawer, and Vogt 2021).
DS reconstruction, however, goes beyond forecasting in that we also require an approximation to the governing equations which captures the temporal and geometrical structure of the original system.
Such a model enables further analysis and mechanistic insight into the underlying dynamics (Strogatz 2015).
Kramer et al.
(2022) recently proposed a nonlinear state space model embedded within a sequential VAE for reconstructing DS from multimodal time series data.
Their work demonstrates the advantages of exploiting various data modalities simultaneously for DS reconstruction, including non-Gaussian like categorical series.
Here we develop a more efficient approach that takes into account recent insights on training RNNs for DS reconstruction (Mikhaeil, Monfared, and Durstewitz 2022).
Method
Our approach to DS reconstruction from multimodal time series rests on three components: 1) A specific type of dynamically interpretable and mathematically tractable RNN, the recently introduced ‘dendritic PLRNN (dendPLRNN)’
(Brenner et al. 2022); 2) a specific TF algorithm, sparse identityTF, for guiding the training process (Brenner et al. 2022;
Mikhaeil, Monfared, and Durstewitz 2022); 3) an MVAE for producing a multimodal TF signal, trained jointly with the dendPLRNN through a combined loss.
The whole procedure is illustated in Fig.
dendPLRNN
The dendPLRNN (Brenner et al. 2022) used for DS reconstruction is defined by the M -dimensional latent process equation
[Equation]
which describes the temporal evolution of a M -dimensional latent state vector z t
= ( z 1 t . . .
z Mt ) T , with a linear diagonal matrix term A ∈ R M × M , off-diagonal matrix W ∈ R M × M , and diagonal noise covariance matrix Σ ∈ R M × M .
The nonlinearity is given by the ‘dendritic’ spline expansion
[Equation]
with slopes α b ∈ R and thresholds h b ∈ R M (Brenner et al. 2022).
This formulation retains analytical access to fixed points and cycles (Durstewitz 2017) and allows for a translation into an equivalent continuous-time representation (Monfared and Durstewitz 2020).
To infer the latent process equation jointly from multiple data modalities, the dendPLRNN is connected to different decoder models that take the distinct distributional properties of each modality into account (see next section and Appx.).
For example, for normally distributed data this may take the simple linear
Gaussian form
[Equation]
with factor loading matrix B ∈ R N × M , and Gaussian observation noise η t ∼ N ( 0 , Γ ) with diagonal covariance Γ ∈ R N × N .
Identity Teacher Forcing Without any purpose-tailoring of the RNN architecture for dealing with exploding gradients, sparse TF (Mikhaeil, Monfared, and Durstewitz 2022), a variant of TF (Williams and Zipser 1989; Pearlmutter 1990), can minimize the problem when training on chaotic systems.
It does so by balancing loss and trajectory divergence with the need to capture relevant long time scales.
To apply sparse
TF to the latent space, the observation model needs to be inverted.
Most simply, this is achieved if we have a linearGaussian model as in Eq.
and choose an “identity-mapping” for the observation model ˆ x t
= I
z t , where I ∈ R N × M with I kk = 1 if k ≤ N and zeroes everywhere else.
Consider an observed time series X
= { x 1 , x 2 , · · · , x T } generated by a DS we want to reconstruct.
For times lτ + 1 , l ∈ N 0 , with forcing interval τ ≥ 1 , we replace the first N latent states by observations ˆ z k,lτ +
= x k,lτ +1 , k ≤ N .
The observations are thereby mapped one-to-one onto a subset of latent ‘readout’ states, while the remaining latent states, ˆ z k,lτ +
=
z k,lτ +1 , k > N , remain unaffected.
Replacing latent states with observations significantly helps to stabilize training while allowing unimpeded gradient flow through the non-forced states.
As shown in Mikhaeil, Monfared, and Durstewitz (2022), the best tradeoff between exploding gradients and capturing relevant long-term dependencies is achieved when choosing the forcing interval τ according to the system’s maximal Lyapunov exponent.
However, for non-continuous data (like counts), this cannot readily be determined.
In this case the forcing interval τ may also be considered as a hyperparameter, and optimal settings found via grid search.
With F = { lτ + 1 } l ∈ N 0 , the dendPLRNN updates can then be written as
[Equation]
The squared error loss used within the BPTT-TF algorithm is given by L MSE =
(cid:80) Tt =
∥ x t − I z t ∥ 22 and is calculated prior to forcing for every time step.
When sampling from the trained model, the dendPLRNN runs freely without forcing.
Multimodal Variational Autoencoder (MVAE)
While our model formulation is general and can work with any combination of data modalities, for the present exposition we consider time series of multivariate Gaussian, ordinal, and count nature of length T , Y = {{ x 1 , · · · , x T } ; { o 1 , · · · , o T } ; { c 1 , · · · , c T }} .
We employ an MVAE for inferring a joint latent representation over these data.
We denote encoded states at time t by ˜ z t ∈ R K to avoid confusion with the latent dynamical process z t ∈ R M generated by the dendPLRNN.
To this end, we minimize the negative Evidence Lower Bound (ELBO)
[Equation]
using the reparameterization trick for latent random variables (Kingma and Welling 2014).
For the encoding step, we concatenate all input modalities and use convolutional neural networks (CNNs) for parameterizing the encoder model q ϕ
( ˜ Z | Y ) , with hyperparameters as proposed in Brenner et al.
This allows the recognition model to embed temporal context into the latent representation (see, e.g., Cui, Chen, and Chen (2016)).
As the latent dendPLRNN process itself is conditionally Gaussian, we make a Gaussian assumption for the variational density q ϕ
( ˜ Z | Y ) =
N ( µ ϕ ( Y ) , Σ ϕ ( Y )) to approximate the true posterior p ( ˜ Z | Y ) , where mean and covariance are functions of the data.
We further use the common mean field approximation to factorize q ϕ
( ˜ Z | Y ) across time (Girin et al. 2021).
Observations are assumed to be conditionally independent given the latent states ˜ z t .
A linear decoder as in Eq.
3 is employed for Gaussian data, a cumulative link model for ordinal data, and a log-link function for Poisson data (but recall the present framework can deal with any set of distributional models):
[Equation]
Due to the conditional independence given the latent states, the likelihood terms related to the observations simply sum up,
[Equation]
Multimodal Teacher Forcing The MVAE provides a flexible and convenient framework for mapping any combination of observed data modalities into the process model’s latent space, and is employed here to generate a TF signal for the dendPLRNN.
We call this the MVAE-TF approach.
For training, observed time series Y = { y 1 , · · · , y T } from possibly many different and non-Gaussian modalities are first converted to a common latent code ˜ Z
= { ˜ z 1 , · · · , ˜ z T } using the MVAE.
The initial condition z 1 of the latent process is inferred from the encoded state ˜ z
If K < M , the M − K remaining states are randomly sampled from a standard normal distribution.
The dendPLRNN is then iterated forward from z 1 across the length of the observed time series T to obtain a latent path Z
= { z 1 , z 2 , · · · , z T } , while in analogy to identity-TF at times lτ +1 , l ∈ N 0 , the first K latent states are replaced by the encoded states z k,
lτ +1 =
˜
z k,lτ +1 , k ≤ K .
The first K states of the generated latent trajectory Z (using the unforced states) are then used to compute the modalityspecific negative log-likelihoods for the observed multimodal time series, L PLRNN =
− (cid:80) Tt =1 (log p θ ( x t | z 1: K,t ) + log p θ ( o t | z 1: K,t ) + log p θ ( c t | z 1: K,t )) , using the decoder models from Eq.
6 (hence, importantly, the latent states z t and ˜ z t of the dendPLRNN and MVAE, respectively, are both coupled to the observations through the very same set of decoder models with same parameters).
A crucial assumption of the MVAE-TF framework now is that the process prior p θ ( ˜ Z ) in Eq.
5 comes from the dendPLRNN.
For the second term in Eq.
5 we assume
[Equation]
where the expectation value is approximated by L Monte
Carlo samples ˜
z ( l ) t ∼ q ϕ
( ˜ z t ) .
To connect the latent codes of the MVAE and the dendPLRNN, the model prior of the
MVAE is instantiated through the dendPLRNN by taking µ t =
z 1: K,t (i.e., the first K states of the generated latent sequence { z t } ).
As the initial state z 1 is estimated directly from the encoded state ˜ z 1 , the term for t = 1 evaluates to zero.
Setting L = 1 , we thus obtain a consistency loss between encoded and generated latent state paths as
L cons =
T (cid:88)
[Equation]
The general scheme for the MVAE-TF is visualised in Fig.
The total MVAE-TF loss is thus given by the reconstruction loss of the MVAE (first and third term in Eq. 5), the latent loss in Eq. 8 that ensures consistency between the latent codes of the MVAE and dendPLRNN, and the dendPLRNN loss from the likelihoods of the observed time series Y given the predicted latent path Z (above Eq.
[Equation]
Experiments
Performance Measures In DS reconstruction, we are primarily interested in capturing invariant properties of the underlying DS like its geometrical and asymptotic temporal structure.
While mean-squared errors (MSE) may be used to asses short-term ahead prediction, especially in chaotic systems they are not suited for evaluating the system’s longerterm behavior because of exponential trajectory divergence (Koppe et al.
2019b; Mikhaeil, Monfared, and Durstewitz
To assess the quality of reconstructions, we compile a set of general as well as modality-specific performance measures (for more details
, see Appendix): • To assess overlap in attractor geometries , we employ a Kullback-Leibler divergence in state space, D stsp (Koppe et al. 2019b; Brenner et al. 2022).
• To assess the agreement in asymptotic temporal structure , in the continuous-Gaussian case power spectra were first computed through the Fast Fourier Transform (Cooley and Tukey 1965) on all observed time series dimensions and slightly smoothed with Gaussian kernels to remove noise.
The Hellinger distance D H between empirical and model-generated power spectra was then computed.
To assess the agreement in temporal structure for ordinal and count data , we determined the MSE between autocorrelation functions (Wiener 1930) computed for up to 200 time lags based on the Spearman rank correlation coefficient for discrete ordinal data (see Appx.
Fig.
9, OACF for ordinal and CACF for count data in Table 1).
We also assessed how well the Spearman cross-correlation structure between observed ordinal variables was preserved within the generated time series (SCC, see Appx.
Fig. 4).
• To assess short-term behavior , for Gaussian data the classical 10-step-ahead prediction error (PE) along test set trajectories was used.
For ordinal data , we computed a linear L 1 PE (OPE) instead, as suggested in ¨ O˘gretir et al.
Benchmark Comparisons on Multi-Modal Time Series
We first evaluate the MVAE-TF’s ability to combine different
[Table]
for inferring a common latent DS model on three ground-truth datasets, and compare its performance to a variety of other methods.
We generated a training and a test set of 100 ,time steps from a Lorenz-63 and a R ¨ ossler system with 1% process noise, and a 6 d Lewis-Glass network model (Lewis and Glass 1992; Gilpin 2022), all in their chaotic regimes (see Appx.
for detailed parameter settings and numerical integration).
From the simulated trajectories we then sample ordinal and count observations using Eqs.
18 and 20 (with randomly drawn parameters), as well as continuous observations with 10% Gaussian noise.
Example reconstructions of the MVAE-TF are in Fig.
For comparison, the same dendPLRNN was trained as proposed in Kramer et al.
with a sequential VAE, optimizing its multimodal ELBO.
This to our knowledge is currently the only other approach specifically designed for DS reconstruction from various data modalities observed simultaneously.
Results are given in Table 1.
We also included three other comparisons: One naive approach is to transform all data modalities to approximately
Gaussian (via Box-Cox & Gaussian kernel smoothing, see
Appx.), and then either train the dendPLRNN via standard
BPTT-TF (Brenner et al. 2022) or by VAE-TF, as proposed here, but without multi-modal integration (labeled GVAE-TF in Table 1).
A third approach that can deal with multi-modal observation models but, unlike TF, does not require model inversion, is ‘multiple shooting (MS)’, a method suggested in the dynamical systems literature (Voss, Timmer, and Kurths (2004); see Appx.
for more details).
As evidenced in Table 1,
MVAE-TF outperforms all these other possible model setups.
We also tested the situation where faithful reconstruction should be possible from the Gaussian modality alone, with just 1% Gaussian noise.
As seen in Appx.
Table 4, even in this case MVAE-TF outperforms all other methods (possibly because here count and ordinal data may actually tend to mislead DS reconstruction for some of the other models).
Challenging Data Situations To really challenge the algorithm, we next tested a scenario where continuous observations from the Lorenz-63 system were heavily distorted by
Gaussian noise with 50% of the data variance.
At the same time, ordinal observations with 8 variables divided intoordered categories each, o nt ∈ { 1 . . .
7 } , n = 1 ... 8 , were sampled using Eq.
We then trained the dendPLRNN via
MVAE-TF once with, and once without, ordinal observations.
Figure 2(a) proves that with ordinal observations on board,
DS reconstruction is, in principle, possible even under these challenging conditions.
The cumulative histograms of the geometric measure D stsp in Figure 2(b) and the temporal measure D H in Figure 8 (b), comparing runs in the unimodal and multimodal settings, furthermore shows that inclusion of ordinal observations significantly improves reconstruction of the underlying system.
Motivated by these results, we pushed the system even further and attempted DS reconstruction solely from ordinal data (created as above), i.e. completely omitting continuous observations.
This is profoundly more challenging than the multimodal setting with Gaussian data, since the ordinal process considerably coarse-grains the underlying continuous dynamical process.
Since in this case we do not have a direct linear mapping between ground truth state space and that of the trained dendPLRNN (which in the case of
Gaussian observations would simply be given by Eq. 3), we construct one post-hoc by optimizing a linear operator given by a linear dimensionality reduction (PCA) concatenated with a geometry-preserving rotation operation (see Appx.).
Fig.shows that, using MVAE-TF, successful DS reconstruction is, in principle, even feasible in this situation.
Comparable results could not be achieved by the multimodal SVAE (see
Appx., Table 3).
Conclusions
In the present work we introduced a novel training method for
DS reconstruction from multimodal time series data based on dynamically interpretable RNNs.
While DS reconstruction is meanwhile a large field in scientific ML (Brunton and Kutz 2022), reconstruction based on multimodal, especially noncontinuous/ non-Gaussian data has hardly been addressed so far, although such scenarios are commonplace in many areas like medicine, neuroscience, or climate research.
Here we utilize recent insights on guiding the training process by control signals (Mikhaeil, Monfared, and Durstewitz 2022) within a novel multimodal data integration framework for
DS reconstruction.
In our approach, a sparse TF signal is generated by an MVAE that translates many different data modalities into a common latent code.
This yields a flexible inference framework for recovering DS from multimodal data while avoiding common training issues associated with chaotic systems (Mikhaeil, Monfared, and Durstewitz 2022).
We show that for various chaotic benchmarks and sampling conditions, assessed by a variety of DS statistics, training the dendPLRNN by MVAE-TF clearly outperformed several other model formulations, including a SVAE-based approach previously suggested by Kramer et al.
We conjecture that this is due to the fact that MVAE-TF allows latent trajectories to evolve freely for longer time spans during training.
In contrast, in classical SVAEs temporal consistency is ensured only through the one-step ahead prediction terms in the ELBO.
Moreover, the MVAE-TF algorithm was able, in principle, to recover the underlying attractor based on ordinal observations alone.
That attractor geometries can be faithfully reconstructed from discrete random variables alone to our knowledge has indeed never been shown before.
A further advantage of our method is that it is modular, that is, subcomponents of the algorithm, such as the encoder or latent model, can easily be replaced within the same overall training framework.
More sophisticated encoder models, e.g. combinations of mixtures-of-experts or products-of-experts (Wu and Goodman 2018; Shi et al. 2019), could be used to find more effective embeddings of multimodal data.
Optimal weighing schemes for the different loss terms making up the total loss (Bakarji et al. 2022) may further improve performance.
While here we presented first results on DS reconstruction for discrete data, whether and how much of the original state space topology of a data-generating DS can be recovered from non-continuous, non-Gaussian random variables remains an important topic for future theoretical and empirical research.
Acknowledgements
This work was funded by the European Union’s Horizonresearch and innovation Programme under grant agreement 945263 (IMMERSE), by the German Research Foundation (DFG) within the collaborative research center TRR- (project A06), and by a living lab grant by the Federal Ministry of Science, Education and Culture (MWK) of the state of
Baden-W¨urttemberg, Germany (grant number 31-7547.2237/3/2), to DD and GK.
Performance Measures
Geometrical Measure To assess the (dis-)agreement D stsp between the data distribution p true ( x ) and the generated distribution p gen ( x | z ) across state space , ˆ p true ( x ) and ˆ p gen ( x | z ) are estimated by sampling 100 trajectories with randomly drawn initial conditions and 1000 time steps each.
Transients are removed from each sampled trajectory to ensure that the measure is evaluated on the limit set.
The match between distributions is then approximated by binning the state space into discrete bins (Koppe et al.
2019b).
[Equation]
Here, K is the total number of bins.
A range of 2 × the data standard deviation on each dimension was partitioned into m bins, leading to a total of K = m N bins, where N is the dimension of the ground truth system.
Due to the exponential scaling of the number of bins with system dimensionality, for the 6 -dimensional Lewis-Glass network model we instead used an approximation of p true ( x ) and p gen ( x | z ) based on Gaussian mixture models placed along trajectories, as described in Brenner et al.
Power
Spectrum Hellinger Distance
The power spectrum Hellinger distance ( D H ) was obtained by first sampling a time series of 100 , 000 time steps and computing dimension-wise Fast Fourier Transforms (using scipy.fft ) for both the ground truth system and simulated time series.
The noise dominated high-frequency tails of the spectra were cut off, and the power spectra were slightly smoothed with a Gaussian kernel and normalized.
We then computed the Hellinger distance (Mikhaeil,
Monfared, and Durstewitz 2022) between smoothed power spectra of ground-truth, F ( ω ) , and generated, G ( ω ) , trajectories given by
[Equation]
The dimension-wise Hellinger distances were then averaged to yield the D H values from Tables 1 and 4.
Mean Squared Prediction Error
A mean squared prediction error (PE) was computed across test sets of length T =
by initializing the trained dendPLRNN with the test set time series up to some time point t , from where it was then iterated forward by n time steps to yield a prediction at time step t + n .
The n -step PE is then defined as the MSE between predicted and true observations:
[Equation]
Due to exponential divergence of initially close trajectories in chaotic systems, the PE is sensible only for a limited number of time steps (Koppe et al. 2019b).
Ordinal Prediction Error
The ordinal PE was computed similarly as the mean squared PE above, but – as pointed out in ¨O ˘ gretir et al.
(2022) – due to the non-metric nature of ordinal data taking the absolute ( L 1 ) deviation between observed and predicted values is more sensible:
[Equation]
Spearman Cross-Correlation (SCC)
To assess whether the global cross-correlation structure between the different ordinal time series is preserved by the reconstruction method, the Spearman correlation between each pair of ordinal time series was computed based on 100 , 000 time steps long samples, using scipy.stats.spearmanr , for both generated and test set data (see Fig. 4).
The squared error between all elements of the correlation matrices was then taken and averaged.
Spearman Autocorrelation Function (SACF)
To assess the temporal agreement between generated and ground truth ordinal and count observations, we computed a measure based on the average SACF.
To this end we first sampled a time series of 100 , 000 time steps and compute the dimension-wise Spearman autocorrelation for time lags up to 200 for both generated data and test set data (see Fig. 9).
The squared error between the resulting SACFs was then averaged across all dimensions.
Geometric reconstruction measure in the absence of continuous observations If the underlying DS was observed only through time series of discrete random variables, we lack a direct mapping between the true and reconstructed continuous state spaces.
To construct a mapping for such cases, we aimed for a linear operator that does not introduce additional degrees of freedom for modifying the reconstructed attractor geometry, but consists only of 1) a projection into a space of same dimensionality (and re-standardization of variables) followed by 2) a (geometry-preserving) rotation.
This was to ensure that the quality of geometrical agreement can be attributed solely to the reconstruction method and not to any post-hoc fitting.
For the first step, we simply used Principal Component Analysis (PCA) to reduce the dendPLRNN’s latent space to the same dimensionality
8 dimension        d i m e n s i o n 8 dimension        d i m e n s i o n
Generated Data 0.
N as that of the ground truth system (which usually is of lower dimensionality).
Afterwards, all axes were re-standardized (as for the original system).
In the second step, a rotation matrix was then determined to rotate the latent state space such as to minimize the same Kullback-Leibler measure D stsp that was used to assess agreement in attractor geometries, see Fig.
5 for an example.
This was done simply by grid search over the space of rotation matrices, as we found numerical optimization to often yield inferior results.
Note that this operation does not alter the geometry of objects in the latent space but merely rotates them such that they are best aligned with their ground truth counterparts (we also attempted Procrustes analysis (Gower 1975), using the Procrustes library (Meng et al. 2022), to determine the best affine mapping between spaces directly, but found this generally to be inferior despite actually being less conservative than our approach).
Comparing different grid- and step sizes in preliminary runs, we fixed parameters such that a single grid search takes no more than 30 − 60 seconds on a single CPU.
To confirm that this procedure yields results in agreement with those obtained from a co-trained linear-Gaussian model fed with continuous observations, we compared D stsp computed in observation space (’ D bin ’) as outlined above with D stsp obtained from ordinal data alone using our PCA+rotation method (’ D PCA ’).
As shown in Fig.
6, these two measures were indeed highly correlated, r ≈ 0 .
An example reconstruction from solely ordinal observations for a R¨ossler system is given in Fig.
For the 6 -dimensional Lewis-Glass chaotic network model, performing a grid search over rotation matrices in the observation space was unfortunately no longer computationally feasible, such that in this case we needed to resort to Procrustes analysis (see above; generally, the Procrustes method aims to superimpose two data sets by optimally translating, rotating, and scaling them, preserving geometric similarity).
In this case, the correlation between the D stsp measures obtained by a co-trained linear model and the one obtained post-hoc via the Procrustes-transformed space dropped to r ≈ 0 . 57 , but was still significant.
Details on Dynamical Systems Benchmarks
Lorenz
63 System The 3 d Lorenz-63 system, originally proposed in Lorenz (1963), is defined by
[Equation]
[Equation]
Parameters used for producing ground truth data in the chaotic regime were σ = 10 , ρ = 28 , and β = 8 / 3 .
Process noise was injected into the system by drawing from a Gaussian term d ϵ ∼ N ( 0 , 0 .
2 d t × I ) .
For both training and test data, a trajectory of 100 , 000 time steps was sampled, performing numerical integration with scipy.odeint ( d t = 0 .
05 ; note this value differs from the one used in Mikhaeil, Monfared, and Durstewitz (2022), explaining the different τ values required).
To obtain multimodal observations, trajectories drawn from the ground truth system were fed into the different types of observation models in Eq. 6, with randomly drawn parameters.
[Figure]
R¨ossler System
The R ¨ ossler system was introduced in R ¨ ossler (1976) as a simplified version of the Lorenz system, and is given by
[Equation]
Parameters used for producing ground truth data in the chaotic regime were a = 0 .
2 , b = 0 .
2 , and c = 5 . 7 .
Process noise was added by drawing d ϵ ∼ N ( 0 , 0 .
2 d t × I ) .
Training and test data was sampled as described above for the Lorenz-63 system, using d t = 0 .
Lewis-Glass Chaotic Network Model We simulate a 6 -dimensional model of a neural network, originally introduced in Lewis and Glass (1992).
Here the individual units of the network are endowed with a continuous gain function G ( x ) =
1+tanh( − αx ), with the vector field given by
[Equation]
To sample from this system in the chaotic regime, we used the Hopfield model implementation in the Python package dysts.flows , based on Gilpin (2022).
Here, α = − 1 , β = 0 . 5 , ϵ = 10 , τ = 2 . 5 , and
K =  
[Equation] [Equation]
[Equation] [Equation]
− 1 −  − − 1 − 1 −   − 1 − −   − 1 − 1 −  

We generated training and test data by maketrajectory , and down-sampled the generated data by a factor of 30 .
We sampled ordinal and count data in the same way as for the other datasets.
Example time series and reconstructions are displayed in Figure 7.
Observation Models for Multimodal Data
Ordinal Model Ordinal data are not associated with a metric space, but there is a natural ordering between variables, as e.g. in survey data in economy or psychology commonly assessed through Likert scales (Likert 1932).
Treating ordinal data as metric can lead to a variety of problems, as pointed out in (Liddell and Kruschke 2018).
Ordinal observations are coupled to latent states via a generalized linear model (McCullagh 1980).
Here, specifically, we assume that the ordinal observations o t are derived from an underlying continuous variable u it , which is linked to the latent states z t via a linear model
[Equation]
where β Ti ∈ R M are the model parameters and ϵ it is an independently distributed Gaussian noise term.
Inverting the link function leads to an expression for the cumulative probabilities:
[Equation]
The probability masses p ( u it = k | z t ) follow from the cumulative distribution via p ( u it = k | z t ) =
p ( u it ≤ k | z t ) − p ( u it ≤ k − 1 | z t ) , from which we can compute the log-likelihood as
[Equation]
Poisson Model For count observations { c t } Tt
=1 , with c t = ( c 1 t , . .
c Lt ) T , we employ a Poisson observation model.
The probability of an observed count under a Poisson model is given by
[Equation]
The probability is related to the latent states via a log-link function by log λ lt = γ ( l )
+ (cid:80)
Mm =1 γ ( l ) m z mt , where γ ( l ) is a 1x M vector.
Thus, λ lt = e γ ( l )
+ γ ( l ) z t is the expected count for the l th observation variable at time t .
Details on Training
Multiple Shooting Another technique from the dynamical systems literature for controlling trajectory divergence, similar in spirit to teacher forcing, is ‘multiple shooting’ (Bock and Plitt 1984).
Multiple shooting aims to solve boundary value problems by dividing time into sub-intervals, treating each subinterval as a separate initial value problem, and then imposing continuity conditions between intervals.
Multiple shooting has also been applied to DS reconstruction, where the initial conditions (‘shooting nodes’) for each interval are model parameters and continuity across intervals is enforced through a penalty term in the loss (Voss,
Timmer, and Kurths 2004).
Hence, rather than controlling trajectory flows through a sparse teacher forcing signal applied after forcing intervals τ , alternatively one may reset the latent model trajectory to an inferred initial condition after τ time steps.
The advantage is that this method does not require inversion of observation models and is hence naturally suited to handle different data modalities without further care (i.e., retaining the distributional properties of the original data).
More specifically, the observed time series Y is partitioned into N seq subsequences Y s , s = 1 . . .
N seq , of length L , and for each subsequence a new initial condition µ s 0 is learned.
During training trajectories are freely generated for L time steps from µ s 0 for each subsequence, and likelihoods for the observed trajectories Y s are computed using the observation models from Eq. 6.
A consistency (penalty) term in the loss ensures continuity between subsequences according to
[Equation]
[Equation]
where F θ in our case is the dendPLRNN, Eq. 1, λ MS is a regularization parameter, and F θ ( z sL ) =
F θ ( F θ ( . . .
F θ (( µ s 0 ))) =
F Lθ ( µ s 0 ) .
The sequence length L plays a similar role as the teacher forcing interval τ for MVAE-TF, controlling the times at which states and gradients are reset during training.
Indeed, optimal settings for τ and L closely agreed for the datasets studied here (see Table 2).
Standard Unimodal Approach with Data ‘Gaussianization’ A naive approach for handling multi-modal observations with any type of DS reconstruction model would be to pre-process all modalities such as to bring them into approximate agreement with Gaussian assumptions.
Thus, for training the dendPLRNN with standard BPTT-TF (Brenner et al. 2022) and GVAE-TF, we transformed ordinal and count observations into approximately Gaussian variables through a Box-Cox-transformation (Box and Cox 1964), z-scoring, and Gaussian kernel smoothing across the time series.
For the optimal width of the Gaussian kernel, we performed a grid search over kernel sizes ν ∈ { 0 , 0 . 01 , 0 .
Optimal settings for the results displayed in
Tables 1 and 4 are given in Table 2.
Hyperparameter Settings To train the dendPLRNN with MVAE-TF, RAdam (Liu et al. 2020) was used with a learning rate scheduler that iteratively reduced the learning rate from 10 − 3 to 10 − 5 during training.
For each epoch, we randomly sampled sequences of length T seq = 300 from the total training data with a batch size of 16 .
The network weights A , W and h from Eq.
1 were initialized according to Talathi and Vartak (2016).
To train the dendPLRNN with the SVAE from Kramer et al.
(2022), we followed the implementation of the encoder model as provided on https://github.com/DurstewitzLab/mmPLRNN, training with a sequence-length of 150 time steps per batch, a hidden dimension of 20 , and other model parameters similar to the runs with the
MVAE-TF.
Hyperparameter settings were chosen such as to approximately keep the number of total parameters similar to that used for MVAE-TF.
[Table] [Table]
[Table] [Table]