Neural Latent Dynamics Models
Abstract
We introduce Neural Latent Dynamics Models (NLDMs), a neural ordinary differential equations (ODEs)-based architecture to perform end-to-end nonlinear latent dynamics discovery, without the need to include any inductive bias related to either the underlying physical model or the latent coordinates space.
The effectiveness of this strategy is experimentally tested in the framework of reduced order modeling, considering a set of problems involving high-dimensional data generated from nonlinear time-dependent parameterized partial differential equations (PDEs) simulations, where we aim at performing extrapolation in time, to forecast the PDE solution out of the time interval and/or the parameters range where training data were acquired.
Results highlight NLDMs’ capabilities to perform low-dimensional latent dynamics learning in three different scenarios.
Introduction
The progress in the field of data-driven model discovery has been recently pushed by the adoption of deep learning-based methods (Raissi [2018], Champion et al.
[2019]), by relying on autoencoders architectures to perform nonlinear dimensionality reduction; this latter represents a key step in view of learning low-dimensional latent coordinates systems.
Recent advancements in the field of reduced order modeling also started to consider deep learning methods, with the introduction of hybrid techniques (Fresca and Manzoni [2022]), combining standard methods such as proper orthogonal decomposition (POD), together with autoencoders, allowing to overcome the main limitations of traditional projection-based techniques, related to their poor performances when modeling nonlinear phenomena.
Classical model discovery methods usually require assumptions on the involved coordinates, prior knowledge about the underlying physical model, typically enforced via specific formulations of the loss function, and/or libraries of candidate functions and their derivatives to perform sparse regression on (Rudy et al.
[2017], Chen et al.
All these assumptions allow to shed light on model interpretability but might be difficult to fulfill in real-world scenarios due to data scarcity, or because they imply infeasible computational loads.
The concept of neural ODEs (Chen et al.
[2018], Kidger [2022]) represents a bridge between dynamical systems and deep learning models, naturally leading to the formulation of latent continuous dynamics discovery frameworks.
In this contribution we propose Neural Latent Dynamics Models (NLDMs), a neural ODEs-based continuous-time architecture, to perform end-to-end latent dynamics learning, as first-order ODEs, without the need to incorporate any inductive bias.
We test the proposed architecture in reduced order modeling settings, applying the framework to multi-dimensional, linear and nonlinear, time-dependent parameterized PDEs, such as a 1D Burgers’ equation, a 2D heat equation, and a 3D nonlinear parameterized elastodynamics problem.
Methods
In this section we describe the general formulation of Neural Latent Dynamics Models (NLDMs) and the architecture for their employment in a reduced order modeling setting.
2.1 Neural Latent Dynamics Models
NLDMs combine autoencoders architecture together with neural ODEs to learn the latent dynamics of the observed time-dependent data.
Recently, multiple time-series modeling approaches combining autoencoders and neural ODEs have been proposed, like generative latent ODEs (Chen et al.
[2018]) and ODE-RNNs (Rubanova et al.
Differently from these strategies, our approach is nonvariational, due to the need to perform deterministic dimensionality reduction as a parallel end goal, and tries to decouple the dimensionality reduction stage from the model discovery one, by defining an architecture made by (i) an encoder, (ii) a latent neural ODE, and (iii) a decoder.
The proposed architecture is presented in a general setting, since it can also be employed to perform standard time-series modeling, by using the latent dynamics module only for time advancements, and disregarding any interest in performing latent model discovery.
The general NLDMs framework employs three learnable functions, an encoder E , a decoder D , and a latent dynamics function f depending on a vector of parameters θ , to model
the observable high-dimensional state u ( t ) ∈
R m evolution, given the initial state u (0) , as follows:
[Equation]
Steps (1) and (3) regard the dimensionality reduction task, via the encoder and the decoder, respectively, while step (2) represents the integration of the latent dynamics function f , linked to the time evolution of the latent state z ( t ) ∈ R n , with n ≪ m .
Operating with discrete observations, the integration step (2) is solved numerically via iterative methods, considering a discretization t 0 , ..., t N of the time interval
[Equation]
with the corresponding formulation of the decoding step (3) that reads as
[Equation]
The training procedure consists of minimizing a loss function given by sum of the squared L 2 -errors between the observed and learned trajectories,
[Equation]
given a dataset of N + 1 sequential observations { t i , u ( t i ) } Ni =0 , from which M different subtrajectories { u ( t j 0 ) , ..., u ( t jN ∗ ) } Mj =1 of length N ∗ + 1 have been sampled, with N ∗ ≤ N .
2.2 NLDMs for Reduced Order Modeling
Regarding the way NLDMs can be exploited in a reduced order modeling setting, our approach builds up from Fresca and Manzoni [2022], since our aim is to keep a modular structure of the overall model, leading to more flexibility when dealing with high-dimensional data, allowing to eventually employ pre-trained sub-modules.
Considering a generic nonlinear parameterized time-dependent PDE, the associated full-order model (FOM) obtained after space discretization can be expressed in the form of a nonlinear parameterized dynamical system that reads as
[Equation]
where u h ( t ; µ ) ∈ R N h represents the time-dependent parameterized solution, M( µ ) ∈ R N h × N h is the mass matrix
P is the input parameters set to which µ belongs, while N h denotes the FOM dimension, related to the space discretization parameter h > 0 .
Formulation (7) justifies the search for a nonlinear latent dynamics describing the evolution of u h ( t ; µ ) .
A common requirement, while performing dimensionality reduction of time-dependent parameterized
PDEs, is to include the information coming from the FOM parameters into the reduced order model (ROM).
As shown in Figure 1, in our setting this is achieved by employing a fusion network F , to combine the concatenated information, previously encoded by the state encoder E u and the parameters encoder E µ , into the initial latent state z ( t 0 ; µ ) ∈ R n , input to the latent neural ODE, where n represents the dimensionality of the reduced manifold.
Both the parameters encoder E µ and the fusion network F are feed-forward neural networks, while the state encoder E u and decoder
D u architectures could differ, depending on the nature of the problem and the complexity of the observed state.
Moreover, a hybrid dimensionality reduction technique has been adopted to deal with high-dimensional FOMs states, by employing the POD basis matrix U extracted from a snapshots matrix S =
[ u h ( t 0 ; µ ) | · · · | u h ( t N ; µ )] , through randomized singular value decomposition (SVD).
Numerical Experiments
NLDMs capabilities have been tested in a reduced order modeling framework, on datasets coming from PDEs discretizations, since they usually involve complex time and spatial-dependent dynamics.
Burgers’ Equation The first test case, inspired by Raissi [2018], consists of modeling the dynamics of the following 1D Burgers’ equation
[Equation]
x ∈ ( − 8 , 8) , t ∈ [0 , 10] (8) discretized with N h = 256 elements in space, over N t = 200 time steps, for a fixed instance of the parameter ν = 0 .
The model’s architecture followed the structure depicted in Figure 1, with a latent state dimension n = 2 , and a latent dynamics function consisting of one hidden layer with 32 units and tanh activation.
The integration was performed by using fifth-order Dormand-Prince adaptive-step solver, from torchdiffeq library (Chen [2018]).
As shown in Figure 2, the method is able to capture the dynamics with a relative error of 6.33e-07 in the training time interval [0,7.5)s, and is able to extrapolate in time, on the remaining test interval [7.5,10]s, with a relative error of 4.95e-06.
Moreover, time extrapolation capabilities have been tested over a much larger timespan, extending the interval to [0,40]s, achieving an overall relative error of 8.35e-05.
The adoption of adaptive-step solvers improve NLDM’s stability and extrapolation capabilities over longer time horizons, at the cost of a slower training procedure.
Heat Equation
The second test concerns modeling the latent dynamics of a 2D heat equation, for a fixed instance of thermal diffusivity α =[-
-]
[Equation]
with a FOM dimension of N h = 64 2 , and N t = 400 .
Since this example involves a 2D domain, the state encoder E u and decoder D u both feature a convolutional architecture.
The adopted latent state dimension is n = 2 , with a latent dynamics function consisting of two hidden layers with 256 units each, and tanh activation, integrated via forward Euler method.
The NLDM learned a two-dimensional dynamics with a mean relative error of 1.81e-07 on the training [0,3)s interval, and of 1.74ewhen performing time extrapolation over the interval [3,4]s, whose last step is shown in Figure 3, together with the latent dynamics evolution over the complete interval.
Parameterized Elastodynamics Problem
The last application regards a 3D parameterized nonlinear elastodynamics problem for a restrained micro-beam subject to internal piezoelectric actuation force, with FOM dimension N h = 7821 , N t = 400 time-steps over the interval [0,25.32]s and 10 instances of the two parameters controlling amplitude and frequency of the piezoelectric oscillating force.
Again, latent state dimension is n = 2 , while the latent dynamics net features one hidden layer with 256 units and tanh activation, integrated via forward Euler method.
The model learned a continuous periodic latent dynamics with a mean relative error of 3.9eon the training interval [0,19)s, and 6.3e-03 when performing time extrapolation considering the whole interval [0,25.32]s, averaged over all the 10 parameter instances.
Moreover, when extrapolating out of the training parameters’ range, the NLDM achieved a mean relative error of 7.8e-03.
Conclusions
We introduced NLDMs, a model architecture combining autoencoders and neural ODEs, to perform end-to-end latent dynamics discovery of high-dimensional systems, without the need of prior assumptions or knowledge about the physics governing the problem. We investigated NLDMs effectiveness in the field of reduced order modeling, by defining a specific encoder structure to deal with parameterized problems, demonstrating their suitability for learning low-dimensional dynamics of nonlinear time-dependent parameterized PDEs systems.