Original: D EEP L ATENT S TATE S PACE M ODELS FOR T IME -S ERIES G ENERATION
Revised: D EEP L ATENT S TATE S PACE M ODELS FOR T IME -S ERIES G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series.
Revised: Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series.

Original: In addition to high computational overhead due to explicitly computing hidden states recurrence, existing ODE–based models fall short in learning sequence data with sharp transitions – common in many realworld systems – due to numerical challenges during optimization.
Revised: In addition to high computational overhead due to explicitly computing hidden states recurrence, existing ODE–based models fall short in learning sequence data with sharp transitions – common in many realworld systems – due to numerical challenges during optimization.

Original: In this work, we propose LS4, a generative model for sequences with latent variables evolving according to a state space ODE to increase modeling capacity.
Revised: In this work, we propose LS4, a generative model for sequences with latent variables evolving according to a state space ODE to increase modeling capacity.

Original: Inspired by recent deep state space models (S4), we achieve speedups by leveraging a convolutional representation of LS4 which bypasses the explicit evaluation of hidden states.
Revised: Inspired by recent deep state space models (S4), we achieve speedups by leveraging a convolutional representation of LS4 which bypasses the explicit evaluation of hidden states.

Original: We show that LS4 significantly outperforms previous continuous-time generative models in terms of marginal distribution, classification, and prediction scores on real-world datasets in the Monash Forecasting Repository, and is capable of modeling highly stochastic data with sharp temporal transitions.
Revised: We show that LS4 significantly outperforms previous continuous-time generative models in terms of marginal distribution, classification, and prediction scores on real-world datasets in the Monash Forecasting Repository, and is capable of modeling highly stochastic data with sharp temporal transitions.

Original: LS sets state–of–the–art for continuous–time latent generative models, with significant improvement of mean squared error and tighter variational lower bounds on irregularly–sampled datasets, while also being × 100 faster than other baselines on long sequences.
Revised: LS sets state–of–the–art for continuous–time latent generative models, with significant improvement of mean squared error and tighter variational lower bounds on irregularly–sampled datasets, while also being × 100 faster than other baselines on long sequences.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Time series are a ubiquitous data modality, and find extensive application in weather (Hersbach et al., 2020) engineering disciplines, biology (Peng et al., 1995), and finance (Poli et al., 2019).
Revised: Time series are a ubiquitous data modality, and find extensive application in weather (Hersbach et al., 2020) engineering disciplines, biology (Peng et al., 1995), and finance (Poli et al., 2019).

Original: The main existing approaches for deep generative learning of temporal data can be broadly categorized into autoregressive (Oord et al., 2016), latent autoencoder models (Chen et al., 2018; Yildiz et al., 2019; Rubanova et al., 2019), normalizing flows (de B ´ ezenac et al., 2020), generative adversarial (Yoon et al., 2019;
Revised: The main existing approaches for deep generative learning of temporal data can be broadly categorized into autoregressive (Oord et al., 2016), latent autoencoder models (Chen et al., 2018; Yildiz et al., 2019; Rubanova et al., 2019), normalizing flows (de B ´ ezenac et al., 2020), generative adversarial (Yoon et al., 2019;

Original: Yu et al., 2022; Brooks et al., 2022), and diffusion (Rasul et al., 2021).
Revised: Yu et al., 2022; Brooks et al., 2022), and diffusion (Rasul et al., 2021).

Original: Among these, continuous-time methods (often based on underlying ODEs) are the preferred approach for irregularly-sampled sequences because they can make predictions at arbitrary time steps and can handle sequences of varying lengths.
Revised: Among these, continuous-time methods (often based on underlying ODEs) are the preferred approach for irregularly-sampled sequences because they can make predictions at arbitrary time steps and can handle sequences of varying lengths.

Original: Unfortunately, existing ODE–based methods (Rubanova et al., 2019; Yildiz et al., 2019) often fall short in learning models for real-world data (e.g., with stiff dynamics) due to their limited expressivity and numerical instabilities during backward gradient computation (Hochreiter, 1998; Niesen & Hall, 2004; Zhuang et al., 2020).
Revised: Unfortunately, existing ODE–based methods (Rubanova et al., 2019; Yildiz et al., 2019) often fall short in learning models for real-world data (e.g., with stiff dynamics) due to their limited expressivity and numerical instabilities during backward gradient computation (Hochreiter, 1998; Niesen & Hall, 2004; Zhuang et al., 2020).

Original: A natural way to increase the flexibility of ODE-based models is to increase the dimensionality of their (deterministic) hidden states.
Revised: A natural way to increase the flexibility of ODE-based models is to increase the dimensionality of their (deterministic) hidden states.

Original: However, that leads to quadratic scaling in the hidden dimensionality due to the need of explicitly computing hidden states by unrolling the underlying recurrence over time, thus preventing scaling to long sequences.
Revised: However, that leads to quadratic scaling in the hidden dimensionality due to the need of explicitly computing hidden states by unrolling the underlying recurrence over time, thus preventing scaling to long sequences.

Original: An alternative approach to increasing modeling capacity is to incorporate stochastic latent variables into the model, a highly successful strategy in generative modeling (Kingma & Welling, 2013; Chung et al., 2015;
Revised: An alternative approach to increasing modeling capacity is to incorporate stochastic latent variables into the model, a highly successful strategy in generative modeling (Kingma & Welling, 2013; Chung et al., 2015;

Original: Song et al., 2020; Ho et al., 2020).
Revised: Song et al., 2020; Ho et al., 2020).

Original: However, this leads to computational costs, and existing models like latent neural ODE models (Rubanova et al., 2019)
Revised: However, this leads to computational costs, and existing models like latent neural ODE models (Rubanova et al., 2019)

Original: inject stochasticity only at the initial condition of the system.
Revised: inject stochasticity only at the initial condition of the system.

Original: In contrast, we introduce LS4, a latent generative model where the sequence of latent variables is represented as the solution of linear state space equations (Chen, 1984).
Revised: In contrast, we introduce LS4, a latent generative model where the sequence of latent variables is represented as the solution of linear state space equations (Chen, 1984).

Original: Unrolling the recurrence equation shows an autoregressive dependence in the sequence of latent variables, the joint of which is highly expressive in representing time series distributions.
Revised: Unrolling the recurrence equation shows an autoregressive dependence in the sequence of latent variables, the joint of which is highly expressive in representing time series distributions.

Original: The high dimensional structure of the latent space, being equivalent to that of the input sequence, allows LS4 to learn expressive latent representations and fit the distribution of sequences produced by a family of dynamical systems, a common setting resulting from non–stationarity.
Revised: The high dimensional structure of the latent space, being equivalent to that of the input sequence, allows LS4 to learn expressive latent representations and fit the distribution of sequences produced by a family of dynamical systems, a common setting resulting from non–stationarity.

Original: We further show how LS4 can learn the dynamics of stiff (Shampine & Thompson, 2007) dynamical systems where previous methods fail to do so.
Revised: We further show how LS4 can learn the dynamics of stiff (Shampine & Thompson, 2007) dynamical systems where previous methods fail to do so.

Original: Inspired by recent works on deep state space models, or stacks of linear state spaces and non-linearities (Gu et al., 2020; 2021), we leverage a convolutional kernel representation to solve the recurrence, bypassing any explicit computations of hidden states via the recurrence equation, which ensures log–linear scaling in both the hidden state space dimensionality as well as sequence length.
Revised: Inspired by recent works on deep state space models, or stacks of linear state spaces and non-linearities (Gu et al., 2020; 2021), we leverage a convolutional kernel representation to solve the recurrence, bypassing any explicit computations of hidden states via the recurrence equation, which ensures log–linear scaling in both the hidden state space dimensionality as well as sequence length.

Original: We validate our method across a variety of time series datasets, benchmarking LS4 against an extensive set of baselines.
Revised: We validate our method across a variety of time series datasets, benchmarking LS4 against an extensive set of baselines.

Original: We propose a set of 3 metrics to measure the quality of generated time series samples and show that LS4 performs significantly better than baselines on datasets with stiff transitions and obtains on average 30% lower MSE scores and ELBO bounds.
Revised: We propose a set of 3 metrics to measure the quality of generated time series samples and show that LS4 performs significantly better than baselines on datasets with stiff transitions and obtains on average 30% lower MSE scores and ELBO.

Original: On sequences with ≈ 20 K lengths, our model trains × 100 faster than the baseline methods.
Revised: On sequences with ≈ 20 K lengths, our model trains × 100 faster than the baseline methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Rapid progress on deep generative modeling of natural language and images has consolidated diffusion (Ho et al., 2020; Song et al., 2020; Song & Ermon, 2019; Sohl-Dickstein et al., 2015) and autoregressive techniques (Brown et al., 2020) as the state–of–the–art.
Revised: Rapid progress on deep generative modeling of natural language and images has consolidated diffusion (Ho et al., 2020; Song et al., 2020; Song & Ermon, 2019; Sohl-Dickstein et al., 2015) and autoregressive techniques (Brown et al., 2020) as the state–of–the–art.

Original: Although various approaches have been proposed for generative modeling of time series and dynamical systems, consensus on the advantages and disadvantages of each method has yet to emerge.
Revised: Although various approaches have been proposed for generative modeling of time series and dynamical systems, consensus on the advantages and disadvantages of each method has yet to emerge.

Original: Deep generative modeling of sequences.
Revised: Deep generative modeling of sequences.

Original: All the major paradigms for deep generative modeling have seen application to time series and sequences.
Revised: All the major paradigms for deep generative modeling have seen application to time series and sequences.

Original: Most relevant to our work are latent continuous– time autoencoder models proposed by Chen et al.
Revised: Most relevant to our work are latent continuous– time autoencoder models proposed by Chen et al.

Original: ; Yildiz et al.
Revised: ; Yildiz et al.

Original: Rubanova et al.
Revised: Rubanova et al.

Original: (2019), where a neural differential equation encoder is used to parametrize as distribution of initial conditions for the decoder.
Revised: (2019), where a neural differential equation encoder is used to parametrize as distribution of initial conditions for the decoder.

Original: Massaroli et al.
Revised: Massaroli et al.

Original: (2021) proposes a variant that parallelizes computation in time by casting solving the ODE as a root finding problem.
Revised: (2021) proposes a variant that parallelizes computation in time by casting solving the ODE as a root finding problem.

Original: Beyond latent models, other continuous– time approaches are given in Kidger et al.
Revised: Beyond latent models, other continuous– time approaches are given in Kidger et al.

Original: (2020), which develops a GAN formulation using SDEs.
Revised: (2020), which develops a GAN formulation using SDEs.

Original: State space models. State space models.
Revised: State space models. State space models.

Original: (SSMs) are at the foundation of dynamical system theory (Chen, 1984) and signal processing (Oppenheim, 1999), and have also been adapted to deep generative modeling.
Revised: (SSMs) are at the foundation of dynamical system theory (Chen, 1984) and signal processing (Oppenheim, 1999), and have also been adapted to deep generative modeling.

Original: Chung et al.
Revised: Chung et al.

Original: ; Bayer & Osendorfer (2014) propose VAE variants of discrete–time RNNs, generalized later by (Franceschi et al., 2020), among others.
Revised: ; Bayer & Osendorfer (2014) propose VAE variants of discrete–time RNNs, generalized later by (Franceschi et al., 2020), among others.

Original: These models all unroll the recurrence and are thus challenging to scale to longer sequences.
Revised: These models all unroll the recurrence and are thus challenging to scale to longer sequences.

Original: Our work is inspired by recent advances in deep architectures built as stacks of linear SSMs, notably
Revised: Our work is inspired by recent advances in deep architectures built as stacks of linear SSMs, notably

Original: S4 (Gu et al., 2021).
Revised: S4 (Gu et al., 2021).

Original: Similar to S4, our generative model leverages the convolutional representation of SSMs during training, thus bypassing the need to materialize the hidden state of each recurrence.
Revised: Similar to S4, our generative model leverages the convolutional representation of SSMs during training, thus bypassing the need to materialize the hidden state of each recurrence.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We briefly introduce relevant details of continuous-time SSMs and their different representations.
Revised: We briefly introduce relevant details of continuous-time SSMs and their different representations.

Original: Then we introduce preliminaries of generative models for sequences.
Revised: Then we introduce preliminaries of generative models for sequences.

Original: 3. S TATE S PACE M ODELS (SSM)
Revised: 3. S TATE S PACE M ODELS (SSM)

Original: A single-input single-output (SISO) linear state space model is defined by the following differential equation
Revised: A single-input single-output (SISO) linear state space model is defined by the following differential equation

Original: [Equation]
Revised: [Equation]

Original: with scalar input x t ∈ R , state h t ∈ R N and scalar output y t ∈ R .
Revised: with scalar input x t ∈ R , state h t ∈ R N and scalar output y t ∈ R .

Original: The system is fully characterized by the matrices A ∈ R N × N ,
Revised: The system is fully characterized by the matrices A ∈ R N × N ,

Original: B ∈ R N × 1 , C ∈ R 1 × N , D ∈ R 1 × 1 .
Revised: B ∈ R N × 1 , C ∈ R 1 × N , D ∈ R 1 × 1 .

Original: Let x, y ∈ C
Revised: Let x, y ∈ C

Original: ([ a, b ] , R ) be absolutely continuous real signals on time interval [ a, b ] .
Revised: ([ a, b ] , R ) be absolutely continuous real signals on time interval [ a, b ] .

Original: Given an initial condition h 0 ∈ R N the
Revised: Given an initial condition h 0 ∈ R N the

Original: SSM (1) realizes a mapping x
Revised: SSM (1) realizes a mapping x

Original: (cid:55)→ y .
Revised: (cid:55)→ y .

Original: SSMs are a common tool for processing continuous input signals.
Revised: SSMs are a common tool for processing continuous input signals.

Original: We consider single input single output (SISO) SSMs, noting that input sequences with more than a single channel can be processed by applying multiple SISO SSMs in parallel, similarly to regular convolutional layers.
Revised: We consider single input single output (SISO) SSMs, noting that input sequences with more than a single channel can be processed by applying multiple SISO SSMs in parallel, similarly to regular convolutional layers.

Original: We use such
Revised: We use such

Original: SSMs as building blocks to map each input dimension to each output dimension in our generative model.
Revised: SSMs as building blocks to map each input dimension to each output dimension in our generative model.

Original: Discrete recurrent representation.
Revised: Discrete recurrent representation.

Original: In practice, continuous input signals are often sampled at time interval ∆ and the sampled sequence is represented by x = ( x t 0 , x t 1 , . . .
Revised: In practice, continuous input signals are often sampled at time interval ∆ and the sampled sequence is represented by x = ( x t 0 , x t 1 , . . .

Original: , x t L ) where t k +

Revised: , x t L ) where t k +

Original: =
Revised: =

Original: t k +∆ .
Revised: t k +∆ .

Original: The discretized SSM follows the recurrence
Revised: The discretized SSM follows the recurrence

Original: [Equation]
Revised: [Equation]

Original: where ¯ A = e A ∆ , ¯ B =
Revised: where ¯ A = e A ∆ , ¯ B =

Original: A − 1 ( e A ∆ − I ) B with the assumption that signals are constant during the sampling interval.
Revised: A − 1 ( e A ∆ − I ) B with the assumption that signals are constant during the sampling interval.

Original: Among many approaches to efficiently computing e A ∆ , Gu et al.
Revised: Among many approaches to efficiently computing e A ∆ , Gu et al.

Original: (2021) use a bilinear transform to estimate e A ∆
Revised: (2021) use a bilinear transform to estimate e A ∆

Original: ≈ ( I − 12 A ∆) − 1 ( I + 12 A ∆) .
Revised: ≈ ( I − 12 A ∆) − 1 ( I + 12 A ∆) .

Original: This recurrence equation can be used to iteratively solve for the next hidden state h t k +1 , allowing the states to be calculated like an RNN or a Neural ODE (Chen et al., 2018; Massaroli et al., 2020).
Revised: This recurrence equation can be used to iteratively solve for the next hidden state h t k +1 , allowing the states to be calculated like an RNN or a Neural ODE (Chen et al., 2018; Massaroli et al., 2020).

Original: Convolutional representation.
Revised: Convolutional representation.

Original: Recurrent representations of SSM are not practical in training because explicit calculation of hidden states for every time step requires O ( N 2 L ) in time and O ( NL ) in space for a sequence of length L 1 .
Revised: Recurrent representations of SSM are not practical in training because explicit calculation of hidden states for every time step requires O ( N 2 L ) in time and O ( NL ) in space for a sequence of length L 1 .

Original: This materialization of hidden states significantly restricts
Revised: This materialization of hidden states significantly restricts

Original: RNN-based methods in scaling to long sequences.
Revised: RNN-based methods in scaling to long sequences.

Original: To efficiently train an SSM, the recurrence equation can be fully unrolled, assuming zero initial hidden states, as
Revised: To efficiently train an SSM, the recurrence equation can be fully unrolled, assuming zero initial hidden states, as

Original: [Equation]
Revised: [Equation]

Original: and more generally as,
Revised: and more generally as,

Original: [Equation]
Revised: [Equation]

Original: For an input sequence x = ( x t 0 , x t 1 , . . .
Revised: For an input sequence x = ( x t 0 , x t 1 , . . .

Original: , x t L ) , one can observe that the output sequence y =
Revised: , x t L ) , one can observe that the output sequence y =

Original: ( y t 0 , y t 1 , . . .
Revised: ( y t 0 , y t 1 , . . .

Original: , y t L ) can be computed using a convolution with a skip connection
Revised: , y t L ) can be computed using a convolution with a skip connection

Original: [Equation]
Revised: [Equation]

Original: This is the well-known connection between SSM and convolution (Oppenheim & Schafer, 1975;
Revised: This is the well-known connection between SSM and convolution (Oppenheim & Schafer, 1975;

Original: Chen, 1984; Chilkuri & Eliasmith, 2021; Romero et al., 2021; Gu et al., 2020; 2021; 2022) and it can be computed very efficiently with a Fast Fourier Transform (FFT), which scales better than explicit matrix multiplication at each step.
Revised: Chen, 1984; Chilkuri & Eliasmith, 2021; Romero et al., 2021; Gu et al., 2020; 2021; 2022) and it can be computed very efficiently with a Fast Fourier Transform (FFT), which scales better than explicit matrix multiplication at each step.

Original: 3. V ARIATIONAL A UTOENCODER (VAE)
Revised: 3. V ARIATIONAL A UTOENCODER (VAE)

Original: VAEs (Kingma & Welling, 2013; Burda et al., 2015) are a highly successful paradigm in learning latent representations of high dimensional data and is remarkably capable at modeling complex distributions.
Revised: VAEs (Kingma & Welling, 2013; Burda et al., 2015) are a highly successful paradigm in learning latent representations of high dimensional data and is remarkably capable at modeling complex distributions.

Original: A VAE introduces a joint probability distribution between a latent variable z and an observed random variable x of the form
Revised: A VAE introduces a joint probability distribution between a latent variable z and an observed random variable x of the form

Original: [Equation]
Revised: [Equation]

Original: where θ represents learnable parameters.
Revised: where θ represents learnable parameters.

Original: The prior p ( z ) over the latent is usually chosen as a standard Gaussian distribution, and the conditional distribution p θ ( x | z ) is defined through a flexible non-linear mapping (such as a neural network) taking z as input.
Revised: The prior p ( z ) over the latent is usually chosen as a standard Gaussian distribution, and the conditional distribution p θ ( x | z ) is defined through a flexible non-linear mapping (such as a neural network) taking z as input.

Original: Such highly flexible non-linear mappings often lead to an intractable posterior p θ ( z | x ) .
Revised: Such highly flexible non-linear mappings often lead to an intractable posterior p θ ( z | x ) .

Original: Therefore, an inference model with parameters ϕ parametrizing q ϕ ( z | x ) is introduced as an approximation which allows learning through a variational lower bound of the marginal likelihood: log
Revised: Therefore, an inference model with parameters ϕ parametrizing q ϕ ( z | x ) is introduced as an approximation which allows learning through a variational lower bound of the marginal likelihood: log

Original: [Equation]
Revised: [Equation]

Original: where D KL ( ·∥· ) is the Kullback-Leibler divergence between two distributions.
Revised: where D KL ( ·∥· ) is the Kullback-Leibler divergence between two distributions.

Original: VAE for sequences.
Revised: VAE for sequences.

Original: Sequence data can be modeled in many different ways since the latent space can be chosen to encode information at different levels of granularity, i.e .
Revised: Sequence data can be modeled in many different ways since the latent space can be chosen to encode information at different levels of granularity, i.e .

Original: z can be a single variable encoding entire trajectories or a sequence of variables of the same length as the trajectories.
Revised: z can be a single variable encoding entire trajectories or a sequence of variables of the same length as the trajectories.

Original: We focus on the latter.
Revised: We focus on the latter.

Original: Given observed sequence variables x ≤ T up to time T discretized into sequence ( x t 0 , . . .
Revised: Given observed sequence variables x ≤ T up to time T discretized into sequence ( x t 0 , . . .

Original: , x t L − 1 ) of length L where t L − 1 = T , a sequence VAE model with parameters θ, λ, ϕ learns a generative and inference distribution
Revised: , x t L − 1 ) of length L where t L − 1 = T , a sequence VAE model with parameters θ, λ, ϕ learns a generative and inference distribution

Original: [Equation]
Revised: [Equation]

Original: where z ≤ t L − 

Revised: where z ≤ t L −

Original: = ( z t 0 , . . .
Revised: = ( z t 0 , . . .

Original: , z t L − 1 ) is the corresponding latent variable sequence.
Revised: , z t L − 1 ) is the corresponding latent variable sequence.

Revised: The approximate posterior q ϕ is explicitly factorized as a product of marginals due to efficiency reasons we shall discuss in the next section.

Original: The variational lowerbound has been considered for discrete sequence data (Chung et al., 2015) with the objective
Revised: Given this form of factorization, the variational lowerbound has been considered for discrete sequence data (Chung et al., 2015) with the objective

Original: [Equation]
Revised: [Equation]

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce Latent S4 (LS4), a latent variable generative model parameterized using
Revised: In this section, we introduce Latent S4 (LS4), a latent variable generative model parameterized using

Original: SSMs.
Revised: SSMs.

Revised: We show how SSMs can parametrize the generative distribution p θ

Original: We show how SSMs can parametrize the generative distribution p θ ( x ≤ T | z ≤ T ) p λ ( z ≤ T ) , the prior distribution p λ ( z ≤ T ) and the inference distribution q ϕ ( z ≤ T | x ≤ T ) effectively.
Revised: [Equation] x ≤ [Equation] ) p λ ( z ≤ T ) , the prior distribution p λ ( z ≤ T ) and the inference distribution q ϕ ( z ≤ T | x ≤ T ) effectively.

Original: For the purpose of exposition, we can assume z t , x t are scalars at any time step t .
Revised: For the purpose of exposition, we can assume z t , x t are scalars at any time step t .

Original: Their generalization to arbitrary dimensions is discussed in Section 4.4.
Revised: Their generalization to arbitrary dimensions is discussed in Section 4.4.

Original: We first define a structured state space model with two input streams and use this as a building block for our generative model.
Revised: We first define a structured state space model with two input streams and use this as a building block for our generative model.

Original: It is an SSM of the form
Revised: It is an SSM of the form

Original: [Equation]
Revised: [Equation]

Original:  where x, y, z ∈ C ([0 , T ] , R ) are continuous real signals on time interval [0 , T ] .
Revised: 6) where x, y, z ∈ C ([0 , T ] , R ) are continuous real signals on time interval [0 , T ] .

Original: We denote
Revised: We denote

Original: H ( x, z, A , B , E , h 0 , t ) =
Revised: H ( x, z, A , B , E , h 0 , t ) =

Original: H β ( x, z, h 0 , t ) , where β denotes trainable parameters A , B , E , as the deterministic function mapping from signals x, z to h t , the state at time t , given initial state h 0 at time 0 .
Revised: H β ( x, z, h 0 , t ) , where β denotes trainable parameters A , B , E , as the deterministic function mapping from signals x, z to h t , the state at time t , given initial state h 0 at time 0 .

Original: The above SSM can be compactly represented by
Revised: The above SSM can be compactly represented by

Original: [Equation]
Revised: [Equation]

Original: When the continuous-time input signals are discretized into discrete-time sequences ( x t 0 , . .
Revised: When the continuous-time input signals are discretized into discrete-time sequences ( x t 0 , . .

Original: , x t L − 1 ) and ( z t 0 , . .
Revised: , x t L − 1 ) and ( z t 0 , . .

Original: z t L ) , the corresponding hidden state at time t k has a convolutional view 
Revised: z t L ) , the corresponding hidden state at time t k has a convolutional view (assuming D = F = 0 for simplicity)

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: (8) which can be evaluated efficiently using FFT.
Revised: (8) which can be evaluated efficiently using FFT.

Revised: Additionally, A is HiPPO-initialized (Gu et al., 2021) for all such SSM blocks.

Original: 4. L ATENT S PACE AS S TRUCTURED S TATE S PACE
Revised: 4. L ATENT S PACE AS S TRUCTURED S TATE S PACE

Original: The goal of the prior model is to realize a sequence of random variables ( z t 0 , z t 1 , . . .
Revised: The goal of the prior model is to realize a sequence of random variables ( z t 0 , z t 1 , . . .

Original: , z t L ) , which the prior distribution p λ ( z ≤ t L ) models autoregressively.
Revised: , z t L ) , which the prior distribution p λ ( z ≤ t L ) models autoregressively.

Original: Suppose ( z t 0 , z t 1 , . . . , z t n ) is a sequence of latent variables up to time t n , we define the prior distribution of z t n autoregressively as
Revised: Suppose ( z t 0 , z t 1 , . . . , z t n ) is a sequence of latent variables up to time t n , we define the prior distribution of z t n autoregressively as

Original: [Equation]
Revised: [Equation]

Original: where the mean µ z,n and standard deviation σ z,n are deterministic functions of previously generated z <t n parameterized by λ .
Revised: where the mean µ z,n and standard deviation σ z,n are deterministic functions of previously generated z <t n parameterized by λ .

Revised: To parameterize the above distribution, we first define an intermediate building block, a stack of which will produce the wanted distribution.

Original: LS4 prior block.
Revised: LS4 prior block.

Original: The forward pass through our SSM is a two–step procedure: first, we consider the latent dynamics of z on [ t 0 , t n − 1 ] where we simply leverage Equation 6 to define the hidden states to follow H β 1 (0 , z, 0 , t ) .
Revised: The forward pass through our SSM is a two–step procedure: first, we consider the latent dynamics of z on [ t 0 , t n − 1 ] where we simply leverage Equation 6 to define the hidden states to follow H β 1 (0 , z, 0 , t ) .

Original: Second, on ( t n − 1 , t n ] , since no additional z is available in this interval, we ignore additional input signals in the ODE and only include the last given latent, i.e .
Revised: Second, on ( t n − 1 , t n ] , since no additional z is available in this interval, we ignore additional input signals in the ODE and only include the last given latent, i.e .

Original: z t n − 1 , as an auxiliary signal for the outputs, which can be compactly denoted, with a final GELU non-linearity, as
Revised: z t n − 1 , as an auxiliary signal for the outputs, which can be compactly denoted, with a final GELU non-linearity, as

Original: [Equation]
Revised: [Equation]

Original: We call this LS4 prior layer , which we use to build our LS4 prior block that is built upon a ResNet structure with a skip connection, denoted as
Revised: We call this LS4 prior layer , which we use to build our LS4 prior block that is built upon a ResNet structure with a skip connection, denoted as

Original: [Equation]
Revised: [Equation]

Original: where ψ denotes the union of parameters β
Revised: where ψ denotes the union of parameters β

Original: i , C y z , F y z , G y z , b y z .
Revised: i , C y z , F y z , G y z , b y z .

Original: We define the final parameters µ z,n and σ z,n for the conditional distribution in the autoregressive model as the result of a stack of
Revised: We define the final parameters µ z,n and σ z,n for the conditional distribution in the autoregressive model as the result of a stack of

Original: LS4 prior blocks .
Revised: LS4 prior blocks .

Original: During generation, as an initial condition, z t 0 ∼ N ( µ z, 0 , σ 2 z, 0 ) where µ z, 0 , σ z,are learnable parameters, and subsequent latent variables are generated autoregressively.
Revised: During generation, as an initial condition, z t 0 ∼ N ( µ z, 0 , σ 2 z, 0 ) where µ z, 0 , σ z,are learnable parameters, and subsequent latent variables are generated autoregressively.

Original: We specify our architecture in Appendix C and use λ to denote the union of all trainable parameters.
Revised: We specify our architecture in Appendix C and use λ to denote the union of all trainable parameters.

Original: 4. G ENERATIVE M ODEL
Revised: 4. G ENERATIVE M ODEL

Original: Given the latent variables, we now specify a decoder that represents the distribution p θ ( x ≤ t [Equation]
Revised: Given the latent variables, we now specify a decoder that represents the distribution p θ ( x ≤ t L | z ≤ t L ) .

Original: Suppose z ≤ t L is a latent path generated via the latent state space model, the output path x ≤ t L also follows the state space formulation.
Revised: Suppose z ≤ t L is a latent path generated via the latent state space model, the output path x ≤ t L also follows the state space formulation.

Original: Assuming we have generated ( x t 0 , . .
Revised: Assuming we have generated ( x t 0 , . .

Original: , x t n − 1 ) and ( z t 0 , . . .
Revised: , x t n − 1 ) and ( z t 0 , . . .

Original: , z t n ) , the conditional distribution of x t n is parametrized as
Revised: , z t n ) , the conditional distribution of x t n is parametrized as

Original: [Equation]
Revised: [Equation]

Original: where σ x is a pre-defined observation standard deviation and µ x,n is a deterministic function of z ≤ t n and x <x n .
Revised: where σ x is a pre-defined observation standard deviation and µ x,n is a deterministic function of z ≤ t n and x <x n .

Original: LS4 generative block.
Revised: LS4 generative block.

Original: Different from the prior block, both observation and latent sequences are input into our model, and we define intermediate outputs g x,n and g z,n as
Revised: Different from the prior block, both observation and latent sequences are input into our model, and we define intermediate outputs g x,n and g z,n as

Original: [Equation]
Revised: [Equation]

Original: which are used to build a LS4 generative block defined as
Revised: which are used to build a LS4 generative block defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where ψ denotes all parameters inside the block.
Revised: 14) where ψ denotes all parameters inside the block.

Original: Note that the generative block gives two streams of outputs, which can be used as inputs for the next stack.
Revised: Note that the generative block gives two streams of outputs, which can be used as inputs for the next stack.

Original: We then define the final mean value µ x,n as the result of a stack of LS4 generative blocks .
Revised: We then define the final mean value µ x,n as the result of a stack of LS4 generative blocks .

Original: The initial condition for generation is given as x t 0 ∼ N ( µ x, 0 ( z 0 , θ ) , σ x ) where µ x, 0 exactly follows our formulation while taking only z t 0 as input.
Revised: The initial condition for generation is given as x t 0 ∼ N ( µ x, 0 ( z 0 , θ ) , σ x ) where µ x, 0 exactly follows our formulation while taking only z t 0 as input.

Original: The subsequent x t n ’s are generated autoregressively.
Revised: The subsequent x t n ’s are generated autoregressively.

Original: We specify our architecture in Appendix C and use θ to denote the union of all trainable parameters.
Revised: We specify our architecture in Appendix C and use θ to denote the union of all trainable parameters.

Original: 4. I NFERENCE MODEL
Revised: 4. I NFERENCE MODEL

Original: The latent variable model up to time t n has intractable posterior p θ ( z ≤ t n | x ≤ t n ) .
Revised: The latent variable model up to time t n has intractable posterior p θ ( z ≤ t n | x ≤ t n ) .

Original: Therefore, we approximate this distribution with q ϕ ( z ≤ t n | x ≤ t n ) using variational inference.
Revised: Therefore, we approximate this distribution with q ϕ ( z ≤ t n | x ≤ t n ) using variational inference.

Original: We parameterize the inference distribution at time t n to depend only on the observed path x ≤ t n :
Revised: We parameterize the inference distribution at time t n to depend only on the observed path x ≤ t n :

Original: [Equation]
Revised: [Equation]

Original: LS4 inference block.
Revised: LS4 inference block.

Original: The inference block is defined as
Revised: The inference block is defined as

Original: [Equation]
Revised: [Equation]

Original: Notice that input x is fully present in [ t 0 , t n ] unlike in the generative model.
Revised: Notice that input x is fully present in [ t 0 , t n ] unlike in the generative model.

Original: ˆ µ z,t and ˆ σ z,t are each a deep stack of our inference blocks similarly defined as before.
Revised: ˆ µ z,t and ˆ σ z,t are each a deep stack of our inference blocks similarly defined as before.

Revised: Here we also justify our choice of factorizing posterior as a product of marginals as in Equation 5.

Revised: By having each z t explicitly depending on x ≤ t n only, we can leverage the fast convolution operation to obtain all z t in parallel, thus achieving fast inference time, unlike the autoregressive nature of the prior and generative model.

Original: 4. LS4: P ROPERTIES AND P RACTICE
Revised: 4. LS4: P ROPERTIES AND P RACTICE

Original: We highlight some properties of LS4.
Revised: We highlight some properties of LS4.

Original: In particular, we compare in the following proposition the expressive power of our generative model against structured state space models.
Revised: In particular, we compare in the following proposition the expressive power of our generative model against structured state space models.

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: (LS4 subsumes S4.)
Revised: (LS4 subsumes S4.)

Original: Given any autoregressive model r ( x ) with conditionals r ( x n | x <n ) parameterized via deep S4 models, there exists a choice of θ, λ, ϕ such that p θ,λ ( x ) =
Revised: Given any autoregressive model r ( x ) with conditionals r ( x n | x <n ) parameterized via deep S4 models, there exists a choice of θ, λ, ϕ such that p θ,λ ( x ) =

Original: r ( x ) and p θ,λ ( z | x ) =
Revised: r ( x ) and p θ,λ ( z | x ) =

Original: q ϕ ( z | x ) , i.e. the variational lower bound (ELBO) is tight.
Revised: q ϕ ( z | x ) , i.e. the variational lower bound (ELBO) is tight.

Original: A proof sketch is provided in Appendix B.
Revised: A proof sketch is provided in Appendix B.

Original: This result shows that LS4 subsumes autoregressive generative models based on vanilla S4 (Gu et al., 2021), given that the architecture between SSM layers is the same.
Revised: This result shows that LS4 subsumes autoregressive generative models based on vanilla S4 (Gu et al., 2021), given that the architecture between SSM layers is the same.

Original: Crucially, with the assumption that we are able to globally optimize the ELBO training objective, LS4 will fit the data at least as well as vanilla S4.
Revised: Crucially, with the assumption that we are able to globally optimize the ELBO training objective, LS4 will fit the data at least as well as vanilla S4.

Original: Scaling to arbitrary feature dimensions.
Revised: Scaling to arbitrary feature dimensions.

Original: So far we have assumed the input and latent signals are real numbers.
Revised: So far we have assumed the input and latent signals are real numbers.

Original: The approach can be scaled to arbitrary dimensions of inputs and latents by constructing LS4 layers for each dimension which are input into a mixing linear layer.
Revised: The approach can be scaled to arbitrary dimensions of inputs and latents by constructing LS4 layers for each dimension which are input into a mixing linear layer.

Original: We call such parallelized SSMs heads and provide a pseudo-code in Appendix C.
Revised: We call such parallelized SSMs heads and provide a pseudo-code in Appendix C.

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: (Efficiency.)
Revised: (Efficiency.)

Original: For a SSM with H heads, an observation sequence of length L and hidden dimension N can be calculated in O ( H ( L + N ) log( L + N )) time and O ( H ( L + N )) space.
Revised: For a SSM with H heads, an observation sequence of length L and hidden dimension N can be calculated in O ( H ( L + N ) log( L + N )) time and O ( H ( L + N )) space.

Original: We provide proof in Appendix B. Note that our model is much more efficient in both time and space than RNN/ODE-based methods (which requires O ( N 2 L ) in time and O ( NL ) in space as discussed in Section 3.1).
Revised: We provide proof in Appendix B. Note that our model is much more efficient in both time and space than RNN/ODE-based methods (which requires O ( N 2 L ) in time and O ( NL ) in space as discussed in Section 3.1).

Original: To demonstrate the computation efficiency, we additionally provide below pseudocode for a single LS4 prior layer 10.
Revised: To demonstrate the computation efficiency, we additionally provide below pseudocode for a single LS4 prior layer 10.

Original: The other blocks can be similarly constructed.
Revised: The other blocks can be similarly constructed.

Original: def LS4_prior_layer(z,
Revised: def LS4_prior_layer(z,

Original: A, B, C, F, h_0): # z: (B, L, 1) K = C @ materialize_kernel(z, A, B, h_0) #
Revised: A, B, C, F, h_0): # z: (B, L, 1) K = C @ materialize_kernel(z, A, B, h_0) #

Original: O((L+N)log(L+N)) time CH = fft_conv(K, z) # O(LlogL) time and O(L)
Revised: O((L+N)log(L+N)) time CH = fft_conv(K, z) # O(LlogL) time and O(L)

Original: space y = gelu(CH + F * z) return y
Revised: space y = gelu(CH + F * z) return y

Original: Note that in practice,  the materialized kernel includes C so that the convolution is computed directly in the projected space,thus bypassing materializing the high-dimensional hidden states.
Revised: Note that in practice, A is HiPPO initialized (Gu et al., 2020) and the materialized kernel includes C so that the convolution is computed directly in the projected space, bypassing materializing the high-dimensional hidden states.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we verify the modeling capability of LS4 empirically.
Revised: In this section, we verify the modeling capability of LS4 empirically.

Original: There are three main questions we seek to answer: (1) How effective is LS4 in modeling stiff sequence data?
Revised: There are three main questions we seek to answer: (1) How effective is LS4 in modeling stiff sequence data?

Original: (2) How expressive is
Revised: (2) How expressive is

Original: LS4 in scaling to real time-series with a variety of temporal dynamics?
Revised: LS4 in scaling to real time-series with a variety of temporal dynamics?

Original: (3) How efficient in training and inference is LS4 in terms of wall-clock time?
Revised: (3) How efficient in training and inference is LS4 in terms of wall-clock time?

Original: 5. L EARNING TO GENERATE DATA FROM STIFF SYSTEMS
Revised: 5. L EARNING TO GENERATE DATA FROM STIFF SYSTEMS

Original: Modeling data generated by dynamics with widely separated time scales has been proven to be particularly challenging for vanilla ODE-based approaches which make use of standard explicit solvers for inference and gradient calculation.
Revised: Modeling data generated by dynamics with widely separated time scales has been proven to be particularly challenging for vanilla ODE-based approaches which make use of standard explicit solvers for inference and gradient calculation.

Original: Kim et al.
Revised: Kim et al.

Original: (2021) showed that as the learned dynamics stiffen up to track data paths, the ODE numerics start to catastrophically fail; the inference cost raises drastically and the gradient estimation process becomes ill–conditioned.
Revised: (2021) showed that as the learned dynamics stiffen up to track data paths, the ODE numerics start to catastrophically fail; the inference cost raises drastically and the gradient estimation process becomes ill–conditioned.

Original: These issues can be mitigated by employing implicit ODE solvers or ad-hoc rescalings of the learned vector field (see (Kim et al., 2021)  details).
Revised: These issues can be mitigated by employing implicit ODE solvers or ad-hoc rescalings of the learned vector field (see (Kim et al., 2021) for further details).

Original: In turn, state–space models do not suffer from stiffness of dynamics as the numerical methods are sidestepped in favor of an exact evaluation of the convolution operator.
Revised: In turn, state–space models do not suffer from stiffness of dynamics as the numerical methods are sidestepped in favor of an exact evaluation of the convolution operator.

Original: We hereby show that LS4 is able to model data generated by a prototype stiff system.
Revised: We hereby show that LS4 is able to model data generated by a prototype stiff system.

Original: FLAME problem We consider a simple model of flame growth (FLAME) (Wanner & Hairer, 1996), which has been extensively studied as a representative of highly stiff systems:
Revised: FLAME problem We consider a simple model of flame growth (FLAME) (Wanner & Hairer, 1996), which has been extensively studied as a representative of highly stiff systems:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ { 3 , 4 , . . . , 10 } .
Revised: where p ∈ { 3 , 4 , . . . , 10 } .

Original: For each p , 1000 trajectories are generated for t ∈ [0 , 1000] with unit increment.
Revised: For each p , 1000 trajectories are generated for t ∈ [0 , 1000] with unit increment.

Original: Generation.
Revised: Generation.

Original: In Figure 1a, we show  that our samples closely match the ground-truth data.The
Revised: In Figure 1a, we show the mean trajectories and the distribution at each time step and that our samples closely match the ground-truth data.

Original:  Latent ODE (Rubanova et al., 2019) instead fails to do so and produces non–stiff samples drastically different from the target.
Revised: The Latent ODE (Rubanova et al., 2019) instead fails to do so and produces non–stiff samples drastically different from the target.

Original: [Figure]
Revised: [Figure]

Original: Marginal Distribution.
Revised: Marginal Distribution.

Original: We plot the marginal distribution of the real data and the generated data from both our model and Latent ODE.
Revised: We plot the marginal distribution of the real data and the generated data from both our model and Latent ODE.

Original: Since the stiff transitions are mostly distributed before 10% of total steps, we visualize the marginal histograms at 4 time steps equally spaced between the 0 .
Revised: Since the stiff transitions are mostly distributed before 10% of total steps, we visualize the marginal histograms at 4 time steps equally spaced between the 0 .

Original: 5% and 10% steps in log scale (see Figure 1b).
Revised: 5% and 10% steps in log scale (see Figure 1b).

Original: We observe that the empirical histogram matches the ground truth distribution significantly better than what is produced by Latent ODEs, as also qualitatively visible from the samples in (a).
Revised: We observe that the empirical histogram matches the ground truth distribution significantly better than what is produced by Latent ODEs, as also qualitatively visible from the samples in (a).

Original: 5. G ENERATION WITH R EAL T IME -S ERIES D ATASETS
Revised: 5. G ENERATION WITH R EAL T IME -S ERIES D ATASETS

Original: We investigate the generative capability of LS4 on real time-series data.
Revised: We investigate the generative capability of LS4 on real time-series data.

Original: We show that our model is capable of fitting a wide variety of time-series data with significantly different temporal dynamics.
Revised: We show that our model is capable of fitting a wide variety of time-series data with significantly different temporal dynamics.

Original: We leave implementation details to Appendix D.1.
Revised: We leave implementation details to Appendix D.1.

Original: Data.
Revised: Data.

Original: We use Monash Time Series Repository (Godahewa et al., 2021), a comprehensive benchmark containing 30 time-series datasets collected in the real world, and we choose FRED-MD, NN5 Daily,
Revised: We use Monash Time Series Repository (Godahewa et al., 2021), a comprehensive benchmark containing 30 time-series datasets collected in the real world, and we choose FRED-MD, NN5 Daily,

Original: Temperature Rain, and Solar Weekly
Revised: Temperature Rain, and Solar Weekly

Original: as our target datasets.
Revised: as our target datasets.

Original: Each dataset exhibits unique temporal dynamics, which makes generative learning a challenging task.
Revised: Each dataset exhibits unique temporal dynamics, which makes generative learning a challenging task.

Original: A sample from each dataset can be visualized in Figure 2.
Revised: A sample from each dataset can be visualized in Figure 2.

Original: Metrics.
Revised: Metrics.

Original: We propose 3 different metrics for measuring generation performance, namely Marginal ,
Revised: We propose 3 different metrics for measuring generation performance, namely Marginal ,

Original: Classification , and Prediction scores.
Revised: Classification , and Prediction scores.

Original: Marginal scores calculate the absolute difference between empirical probability density functions of two distributions – the lower the better (Ni et al., 2020).
Revised: Marginal scores calculate the absolute difference between empirical probability density functions of two distributions – the lower the better (Ni et al., 2020).

Original: Following Kidger et al.
Revised: Following Kidger et al.

Original: (2021), we define Classification scores as using a sequence model to classify whether a sample is real or generated and use its cross-entropy loss as a proxy for generation quality – the higher the less distinguishable the samples, thus better the generation.
Revised: (2021), we define Classification scores as using a sequence model to classify whether a sample is real or generated and use its cross-entropy loss as a proxy for generation quality – the higher the less distinguishable the samples, thus better the generation.

Original: Similarly, Prediction scores use a train-on-synthetic-test-on-real seq2seq model to predict k steps into the future – the lower the more predictable, thus better the generation.
Revised: Similarly, Prediction scores use a train-on-synthetic-test-on-real seq2seq model to predict k steps into the future – the lower the more predictable, thus better the generation.

Original: We use a 1-layer S4 (Gu et al., 2021) for both Classification and Prediction scores (see Appendix D.1 for more details).
Revised: We use a 1-layer S4 (Gu et al., 2021) for both Classification and Prediction scores (see Appendix D.1 for more details).

Original: We will discuss the necessity of all 3 metrics in the following discussion section.
Revised: We will discuss the necessity of all 3 metrics in the following discussion section.

Original: We compare our model with several time-series generative models, namely the VAE-based methods such as RNN-VAE (Rubanova et al., 2019), GP-VAE (Fortuin et al., 2020), ODE 2 VAE (Yildiz et al., 2019), Latent ODE (Rubanova et al., 2019), GAN-based methods such as TimeGAN (Yoon et al., 2019) and SDE GAN (Kidger et al., 2021), and SaShiMi (Goel et al., 2022).
Revised: We compare our model with several time-series generative models, namely the VAE-based methods such as RNN-VAE (Rubanova et al., 2019), GP-VAE (Fortuin et al., 2020), ODE 2 VAE (Yildiz et al., 2019), Latent ODE (Rubanova et al., 2019), GAN-based methods such as TimeGAN (Yoon et al., 2019) and SDE GAN (Kidger et al., 2021), and SaShiMi (Goel et al., 2022).

Original: We show quantitative results in Table 1.
Revised: We show quantitative results in Table 1.

Original: [Table]
Revised: [Table]

Original: Our model significantly outperforms the baselines on all datasets.
Revised: Our model significantly outperforms the baselines on all datasets.

Original: We note that baseline models have a hard time modeling NN5 Daily and Temperature Rain where the transition dynamics are stiff.
Revised: We note that baseline models have a hard time modeling NN5 Daily and Temperature Rain where the transition dynamics are stiff.

Original: For Temperature Rain where most data points lie around x -axis with sharp spikes throughout, latent ODE generates mostly closely to the x -axis, thus achieving lower marginal scores, but its generation is easily distinguishable from data, thus making it a less favorable generative model.
Revised: For Temperature Rain where most data points lie around x -axis with sharp spikes throughout, latent ODE generates mostly closely to the x -axis, thus achieving lower marginal scores, but its generation is easily distinguishable from data, thus making it a less favorable generative model.

Original: We demonstrate that Marginal scores alone are an insufficient metric for generation quality.
Revised: We demonstrate that Marginal scores alone are an insufficient metric for generation quality.

Original: SaShiMi, an autoregressive model based on S4, does not perform as well on time series generation in the tasks considered.
Revised: SaShiMi, an autoregressive model based on S4, does not perform as well on time series generation in the tasks considered.

Original: We further discuss the reason in Appendix D.1.
Revised: We further discuss the reason in Appendix D.1.

Original: 5. I NTERPOLATION & E XTRAPOLATION
Revised: 5. I NTERPOLATION & E XTRAPOLATION

Original: We also show that our model is expressive enough to fit to irregularly-sampled data and performs favorably in terms of interpolation and extrapolation.
Revised: We also show that our model is expressive enough to fit to irregularly-sampled data and performs favorably in terms of interpolation and extrapolation.

Original: Interpolation refers to the task of predicting missing data given a subset of a sequence while extrapolation refers to the task that data is separated into 2 segments each with half the length of the full sequence, and one predicts the latter segment given the former.
Revised: Interpolation refers to the task of predicting missing data given a subset of a sequence while extrapolation refers to the task that data is separated into 2 segments each with half the length of the full sequence, and one predicts the latter segment given the former.

Original: Data.
Revised: Data.

Original: Following Rubanova et al.
Revised: Following Rubanova et al.

Original: ; Schirmer et al.
Revised: ; Schirmer et al.

Original: (2022), we use USHCN and Physionet as our datasets of choice.
Revised: (2022), we use USHCN and Physionet as our datasets of choice.

Original: The United States Historical Climatology Network (USHCN) (Menne et al., 2015) is a climate dataset containing daily measurements form 1,218 weather stations across the
Revised: The United States Historical Climatology Network (USHCN) (Menne et al., 2015) is a climate dataset containing daily measurements form 1,218 weather stations across the

Original: US for precipitation, snowfall, snow depth, minimum and maximum temperature.
Revised: US for precipitation, snowfall, snow depth, minimum and maximum temperature.

Original: Physionet (Silva et al., 2012) is a dataset containing health measurements of 41 signals from 8,000 ICU patients in their first 48 hours.
Revised: Physionet (Silva et al., 2012) is a dataset containing health measurements of 41 signals from 8,000 ICU patients in their first 48 hours.

Original: We follow preprocessing steps of Schirmer et al.
Revised: We follow preprocessing steps of Schirmer et al.

Original: (2022) for training and testing.
Revised: (2022) for training and testing.

Original: Metrics.
Revised: Metrics.

Original: We use mean squared error (MSE) to evaluate both interpolation and extrapolation.
Revised: We use mean squared error (MSE) to evaluate both interpolation and extrapolation.

Original: We compare our model with RNN (Rumelhart et al., 1985), RNN-VAE (Chung et al., 2014;
Revised: We compare our model with RNN (Rumelhart et al., 1985), RNN-VAE (Chung et al., 2014;

Original: Rubanova et al., 2019), ODE-RNN (Rubanova et al., 2019), GRU-D (Rubanova et al., 2019), Latent
Revised: Rubanova et al., 2019), ODE-RNN (Rubanova et al., 2019), GRU-D (Rubanova et al., 2019), Latent

Original: ODE (Chen et al., 2018; Rubanova et al., 2019), and CRU Schirmer et al.
Revised: ODE (Chen et al., 2018; Rubanova et al., 2019), and CRU Schirmer et al.

Original: Results are shown in Table 2.
Revised: Results are shown in Table 2.

Original: [Table]
Revised: [Table]

Original: We observe that our model outperforms previous continuous-time methods.
Revised: We observe that our model outperforms previous continuous-time methods.

Original: Our model performs less well on extrapolation for Physionet compared to ODE-RNN and latent ODE.
Revised: Our model performs less well on extrapolation for Physionet compared to ODE-RNN and latent ODE.

Original: We postulate that this is due to the high variability granted by our latent space.
Revised: We postulate that this is due to the high variability granted by our latent space.

Original: Since new latent variables are generated as we extrapolate, our model generates paths that are more flexible (hence less predictable) than those of Latent ODE, which instead uses a single latent variable to encode an entire path.
Revised: Since new latent variables are generated as we extrapolate, our model generates paths that are more flexible (hence less predictable) than those of Latent ODE, which instead uses a single latent variable to encode an entire path.

Original: We additional observe that our model achieves ELBO of − 669 . 0 and − 250 . 2 on Physionet interpolation and extrapolation tasks respectively.
Revised: We additional observe that our model achieves ELBO of − 669 . 0 and − 250 . 2 on Physionet interpolation and extrapolation tasks respectively.

Original: These bounds are much tighter lower bounds than other VAEbased methods, i.e . RNN-VAE, which reports − 412 .
Revised: These bounds are much tighter lower bounds than other VAEbased methods, i.e . RNN-VAE, which reports − 412 .

Original: 8 and − 220 . 2 , and latent ODE, which reports − 410 . 3 and − 168 .
Revised: 8 and − 220 . 2 , and latent ODE, which reports − 410 . 3 and − 168 .

Original: We leave additional ELBO comparisons in Appendix D.
Revised: We leave additional ELBO comparisons in Appendix D.

Original: 5. R UNTIME
Revised: 5. R UNTIME

Original: We additionally verify the computational efficiency of our model for both training and inference.
Revised: We additionally verify the computational efficiency of our model for both training and inference.

Original: We do so by training and inferring on synthetic data with controlled lengths specified below.
Revised: We do so by training and inferring on synthetic data with controlled lengths specified below.

Original: Data.
Revised: Data.

Original: We create a set of synthetic datasets with lengths { 80 , 320 , 1280 , 5120 , 20480 } to investigate scaling of training/inference time with respect to sequence length.
Revised: We create a set of synthetic datasets with lengths { 80 , 320 , 1280 , 5120 , 20480 } to investigate scaling of training/inference time with respect to sequence length.

Original: Training is done with 100 iterations through the synthetic data, and inference is performed given one batch of synthetic data (see
Revised: Training is done with 100 iterations through the synthetic data, and inference is performed given one batch of synthetic data (see

Original: Appendix D.3).
Revised: Appendix D.3).

Original: Metrics.
Revised: Metrics.

Original: We use wall-clock runtime measured in milliseconds.
Revised: We use wall-clock runtime measured in milliseconds.

Original: Figure 3 shows model runtime in log scale.
Revised: Figure 3 shows model runtime in log scale.

Original: ODE 2 VAE fails to finish training on the last sequence length within a reasonable time frame, so we omit its plot of the last data point.
Revised: ODE 2 VAE fails to finish training on the last sequence length within a reasonable time frame, so we omit its plot of the last data point.

Original: Our model performs consistently and significantly lower than baselines, which are observed to scale linearly with input lengths, and is × 100 faster than baselines in both training and inference on 20480 length.
Revised: Our model performs consistently and significantly lower than baselines, which are observed to scale linearly with input lengths, and is × 100 faster than baselines in both training and inference on 20480 length.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce LS4, a powerful generative model with latent space evolution following a state space
Revised: We introduce LS4, a powerful generative model with latent space evolution following a state space

Original: ODE.
Revised: ODE.

Original: Our model is built with a deep stack of LS4 prior/generative/inference blocks, which are trained via standard sequence VAE objectives.
Revised: Our model is built with a deep stack of LS4 prior/generative/inference blocks, which are trained via standard sequence VAE objectives.

Original: We also show that under specific choices of model parameters, LS4 subsumes autoregressive S4 models.
Revised: We also show that under specific choices of model parameters, LS4 subsumes autoregressive S4 models.

Original: Experimentally, we demonstrate the modeling power of LS4 on datasets with a wide variety of temporal dynamics and show significant improvement in generation/interpolation/extrapolation quality.
Revised: Experimentally, we demonstrate the modeling power of LS4 on datasets with a wide variety of temporal dynamics and show significant improvement in generation/interpolation/extrapolation quality.

Original: In addition, our model shows × 100 speed-up in training and inference time on long sequences.
Revised: In addition, our model shows × 100 speed-up in training and inference time on long sequences.

Original: LS4 demonstrates improved expressivity and computational efficiency, and we believe that it has a further role to play in modeling general time-series sequences.
Revised: LS4 demonstrates improved expressivity and computational efficiency, and we believe that it has a further role to play in modeling general time-series sequences.