Learning Physics Constrained Dynamics Using Autoencoders
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the problem of estimating states ( e.g., position and velocity) andphysical parameters ( e.g., friction, elasticity) from a sequence of observationswhen provided a dynamic equation that describes the behavior of the system.
Thedynamic equation can arise from first principles ( e.g., Newton’s laws) and provideuseful cues for learning, but its physical parameters are unknown.
To address thisproblem, we propose a model that estimates states and physical parameters of thesystem using two main components.
First, an autoencoder compresses a sequenceof observations ( e.g., sensor measurements, pixel images) into a sequence for thestate representation that is consistent with physics by including a simulation of thedynamic equation.
Second, an estimator is coupled with the autoencoder to predictthe values of the physical parameters.
We also theoretically and empirically showthat using Fourier feature mappings improves the generalization of the estimatorin predicting physical parameters compared to raw state sequences when learningfrom high-frequency data.
In our experiments on three visual and one sensormeasurement tasks, our model imposes interpretability on latent states and achievesimproved generalization performance for long-term prediction of system dynamicsover state-of-the-art baselines.
Introduction
Neural networks have become a core computational component in domains such as computer vision[1], natural language processing [2] and deep reinforcement learning [3].
Recent work has shownthat neural networks can exhibit an inductive bias that is often introduced via designing specificstructures [4].
This bias can be used to encode prior task knowledge that helps the network generalizeto unseen data.
For example, convolution neural networks capture the translation invariance of keyimage features.
In this spirit, we develop a structured neural network model that leverages a dynamicequation to estimate both the state of a dynamical system ( e.g., position and velocity) and its physicalparameters ( e.g., friction constants) from a sequence of partial observations.
Knowing the stateand parameters of the system aids learning a control policy and tracking parameter value changes over time.
For instance, for a self-driving car, we want to learn a neural network that estimates the vehicle’s position and velocity from a sequence of egocentric camera images.
The estimated state can then be used in a control policy.
In addition, we want to track physical parameters over time, e.g.,friction coefficients.
This is useful for vehicle maintenance and safety.
We expect that exploiting a neural network, regularized to follow a dynamic equation, will streamline the required data-intensiveoperations and yield improved performance.
Observations can be direct measurements of some system variables ( e.g., accelerations), or take a more complex form ( e.g., images).
We group such observations over specified time windows and also refer to these groupings as observations.
We obtain a compact representation for these observations that permits observation reconstruction using an autoencoder (see Fig. 1(a))
h s } in Fig.
1(a) is a representation of an observation sequence { o s } over a specified time window, and { ˆ o s } is thereconstructed observation sequence.
Learning the autoencoder is both data and computationallyintensive.
In addition, h s may not be physically interpretable and be a “system state.”
Motivated by these observations, we assume a simulator is provided that specifies the physical laws ofthe system.
This model can arise from first principles( e.g., Newton’s laws), but its free parameters θ ( e.g.,masses, lengths) remain to be specified.
Given an initial state and θ , the physics simulator generates a statetrajectory { ˆ x s } consistent with the laws of physics.
To leverage this model, we require an estimator f ( · )that maps a sequence of states { ˜ x s } to an estimateof θ .
We then couple the estimator f and the physicssimulator with the autoencoder as shown in Fig. 1(b).
We train the autoencoder and f ( · ) to minimize the observation reconstruction loss (cid:80) s ∥
o s − ˆ o s ∥ 22 .
Withinthis process, we train the encoder h ( · ) to minimizethe sum of squared state errors: (cid:80) s ∥
˜ x s − ˆ x s ∥ 22 .
Thecomplete model (Fig. 1(b)) is called Autoencoder with Latent Physics (ALPS).
The paper’s contributions are three-fold.
(1) ALPS isthe method that learns to identify the system parameters and the mappings between states and observationsfrom data.
In contrast, conventional system identification ( e.g., tools in Matlab) requires knowing thefunction mapping and identifying the parameters online, which is costly to run.
(2) We show thatone can learn periodic or vibrational behavior in this setting using a Fourier feature of states.
Weevaluate ALPS in three simulated and one real-world tasks.
ALPS can achieve up to 4.8x and 6.3xbetter physical parameter and state prediction accuracy, respectively, over prior approaches.
Related Work
Physics-informed neural networks.
There is growing interest in including a physics prior oralgebraic and logical constraints into neural networks [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29].
For example, [30, 31, 32, 33, 34, 35, 36, 37, 38,39, 40] exploit Lagrangian or Hamiltonian mechanics to learn an energy-conserving system basedon position, momentum, and the derivatives thereof along trajectories.
These works assume thephysical parameters of the system are constant and need not be estimated.
In contrast, we estimatethe physical parameters.
This is important for fault detection and localization, and for safety.
Papers[41, 42, 43, 44] learn a general physical simulation from data, but their model is required to havestate information or a known forward rendering engine to map states to observations.
In contrast, welearn a physics-based autoencoder to estimate states in an unsupervised manner.
Paper [45] also usesan autoencoder with physics to predict parameters.
However, we show that learning state sequencesas in [45] fails to generalize to unseen parameters when the system exhibits high-frequency behavior.
Fourier features for high-frequency data.
[46, 47] show that using Fourier features helps neuralnetworks learn high-frequency content in image regression tasks.
This approach requires specifyingthe Fourier series coefficients and the basis frequencies.
In contrast, the Fourier features in ourmodel are computed from the states, which allows us to predict physical parameters.
In addition, theconcurrent work [48] replaces the self-attention sublayers [49] with a Fourier transformation of theinput word token in natural language processing.
They show that this method is sufficient to capturesemantic relationships in several text classification tasks.
In this paper, we provide a theoreticaljustification for using Fourier features to learn system dynamics with high-frequency data
Problem Formulation
We consider the continuous-time system
[Equation]
where A ∈ R n × n , B ∈ R n × p and x ∈ R n , u ∈ R p , o ∈ R q , denote the state, input, and observation,respectively.
The function g provides a partial observation o of x .
In most situations we have partialknowledge of A and B from physics.
This is true and reasonable in many large-scale industrialapplications ranging from wind turbines to aircraft, where system designers use physics knowledgeto design machines.
Hence we assume mappings A ( · ) :
Θ → R n × n and B ( · ) :
Θ → R n × p , fromphysical parameters θ ∈ Θ to the system matrices
A ( θ ) , B ( θ ) , are given ( i.e., A and B obtain aspecific sparse and parameteric form).
In addition, Eq.
(1) assumes the system dynamics to be linear,but our approach can be extended to other non-linear cases.
In practice, we only have observations at discrete points in time.
For simplicity, we assume theseare equally spaced at times t = 0 , 1 , . . . .
At sample time t, we have the window of observations{ o s } ts = t − τ +1 , where τ is the window length.
We assume the sampling rate satisfies the Nyquist rate.
Our problem can now be stated as follows.
Given functions A ( · ) :
Θ → R n × n , B ( · ) :
Θ → R n × p ,we seek to learn a network that estimates a state sequence { x s } ts = t − τ +1 , physical parameters θ ,and a mapping g ( · ) from a finite sequence of past observations { o s } ts = t − τ +1 and past knowninputs { u s } ts = t − τ +1 .
Once trained, the network can predict states and has learned to adapt physicalparameters to the context without re-training.
Note that even though the network operates with timesampled variables, the physics simulator can be used to predict states at any time.
This problem setupis distinct from that of HNN [33], HGN [36], or Symplectic ODE-Net [35].
In their setups, the modelis only trained on data from a single physical parameter θ , and hence they need to re-train networksfor every new set of physical parameters.
In addition, compared to conventional system identificationapproaches, we do not need to rerun the solver each time–we only need a forward pass of the networkto get the estimation during deployment (testing time).
To make the learning problem well-defined, i.e., to ensure that we can uniquely identify the unknownsystem parameters, we make the following assumption.
Assumption 3.1.
(Identifiability [50]) :
For a sufficiently large τ, the sequence of states and inputs{ ( x s , u s ) } ts = t − τ +1 uniquely specifies the unknown parameter θ .
Parameter identifiability is a well-studied problem in system identification [50, 51, 52, 53, 54, 55].
For example, the (continuous time) system ˙ x = (cid:20) − ( a + b ) b b − c (cid:21) x
+ (cid:20) 10 (cid:21) u with o =
[1 0] x ,where θ = [ a, b, c ] T and g ( · ) is a linear function, satisfies Assumption 3.1 when b ̸ = 0 .
Nework Architecture
The network in Fig. 1(b) is expanded in Fig.
2 into its four parts: an encoder network, a parameterestimator network, a physics simulator, and a decoder network.
The encoder network h ( · ) (from { o s } to { ˜ x s } ).
We consider two types of observations: (1) pixelimages, or (2) direct measurements of some system variables.
For the first case, we use a convolution neural network to compress a pixel observation o into a compact vector embedding z ′ ∈ R d , whichpreserves image features.
For the second case, we use a feedforward network to project an observationo into some higher dimensional space with a vector embedding z ′ .
In addition, to estimating statesfrom these vector embeddings, we need to aggregate { z ′ s } to extract the local and global context of thedynamics.
One approach is to use recurrent neural networks (RNN) ( e.g., Dreamer [56]).
However,
RNNs suffer from vanishing gradient problems and slow computation when processing long-termsequences.
Hence we use a self-attention network [49] to attend to z ′ of greatest importance topredict states and improve efficiency.
Furthermore, to inject a position signal of observations in the sequence, we add a positional encodingp ∈ R d to z ′
( z :=
z ′ + p ), where p are sine and cosine functions of different frequencies (see [49]).
Finally, we stack embeddings over τ steps to form a matrix Z ∈ R τ × d .
The self-attention module can be formulated as querying a dictionary with key-value pairs associatedwith learnable weight matrices W Q ∈ R d × d Q , W K ∈ R d × d K , and
W V ∈ R d × d V ( d Q = d K here):
Attention ( Q , K , V ) =
softmax ( QK T √ d ) V , where Q :=
ZW Q , K :=
ZW K , V :=
ZW V , andthe softmax is taken over the sequence length τ.
To provide multiview of the embedding subspace,we use the multihead variant of the attention module by concatenating each attention head along thesequence axis
[Equation]
, where W O ∈ R Id V × d are learnable matrices, I is the number of heads, and each attention head i hasits own learnable weight matrices
[Equation]
W Ki ∈ R d × d K , and W Vi ∈ R d × d V .
Finally, a feedforward network takes in the Multihead embedding (of size R τ × d ) and produces theparameters of the distribution for each state in the sequence.
The parameters are used to define aposterior distribution over the encoded state ˜ x s ∼ Q ( ·| ˜ o s ) with the prior P (˜ x s ) .
For a translationalcoordinate, the posterior distribution is a Gaussian distribution with a unit Gaussian prior.
Hence thenetwork predicts a mean µ ∈ R n and a standard deviation σ ∈ R n of a Gaussian distribution.
Inaddition, for a rotational coordinate, the posterior distribution is a von Mises (vM) distribution with aunit vM prior.
Similar to a Gaussian distribution, a vM distribution is defined by two parameters: amean µ ∈ R 2 , ∥ µ ∥ 2 = 1 (the angular position (cos φ, sin φ ) ), and a concentration κ ∈ R + aroundµ .
Such parameterization is found useful in practice, e.g., [57].
Appendix E provides details aboutself-attention networks for estimating states.
The parameter estimator network f ( · ) (from { ˜ x s } to θ ).
The parameter estimator networkpredicts physical parameters from state sequences { ˜ x s } .
However, for systems that involve periodic or vibrational behavior, prior work [46] has shown that neural networks fail to capture thehigh-frequency content in the data.
In addition, real-world data often contain noise.
To improvegeneralization and reduce the effect of noise, we do a Fourier transform on each component j of statetrajectories { ˜ x s ( j ) } ts = t − τ +1 to get { ˜ X ω ( j ) } tω = t − τ +1 :
˜ X ω ( j ) :
= (cid:80) tk = t − τ +
˜ x k
( j ) (cid:104)
cos (cid:16) 2 πτ ωk (cid:17) −i · sin (cid:16) 2 π τ ωk (cid:17)(cid:105) .
Note that for the systems that do not have periodic or vibrational behavior, using{ ˜ x s ( j ) } ts = t − τ +1 would be enough.
One may use { ˜ X ω ( j ) } as features for the parameter estimator network to predict physical parameters.
However in Section 5 we will show that by using the neural tangent kernel (NTK) theory [58], whichtreats neural networks as a kernel regression, the resulting kernel matrix of { ˜ X ω ( j ) } does not preservehigh-frequency components in the data.
To solve this, we will show that using the magnitude of the
Fourier features {| ˜ X ω ( j ) |} alleviates the issue.
Hence the parameter estimator network takes in aconcatenation of {| ˜ X ω ( j ) |} from each component of the state and predicts physical parameters θ .
The physics simulator (from ˜ x t − τ +
and θ to { ˆ x s } ).
Given a start state ˜ x t − τ +1 (from theencoder’s first state prediction) and values for the physical parameters θ , we use the neural ordinarydifferential equation (ODE) [59], a differential ODE solver, to generate a simulated state trajectory{ ˆ x s } ts = t − τ +1 : ˆ x t − τ +1 , . . .
ˆ x t − 1 , ˆ x t =
ODESolver(˜ x t − τ +1 , ˙ x
= A ( θ ) x + B ( θ ) u , τ, ∆) , where
ODESolver takes in a start state, an ODE, a window length, and a sampling time interval ∆ .
Notethat
ˆ x t − τ +
=
˜ x
t − τ +1 .
In addition, using an ODE solver allows us to generate an accurate statetrajectory compared to that of RNN as in [56].
The decoder network g ( · ) (from ˆ x s to ˆ o s ).
Finally, the decoder network is either a deconvolutionalnetwork (for image observations) or a feedforward network (for sensor measurements) that takes ineach individual ODE-simulated state ˆ x s and generates a reconstructed observation ˆ o s .
The loss function.
Given a sequence of τ observations, we minimize the following loss function:
[Equation]
The variational autoencoder (VAE) [60] loss is a variational bound on the marginal log-likelihoodof the data.
It is used to train the encoder h, the estimator f, and the decoder g .
Using VAEsavoids learning degenerated solutions and provides stable training of the network over a deterministic network.
In addition, the observation reconstruction loss encourages reconstructed observations{ ˆ o s } to match true observations { o s } , and the state reconstruction loss constrains encoded states{ ˜ x s } to follow simulated states { ˆ x s } generated by physics.
The former is used to train h, f,and g, and the latter is used to train f and h. Both observation and state reconstruction losses areimportant for training.
We find that removing the state reconstruction term reduces the state predictionperformance of the encoder since the network can predict arbitrary sequences.
In addition, removingthe observation reconstruction term impedes the image reconstruction quality, which is vital fortraining the whole model.
In practice, we tune the weight for each loss term to accommodate differentscales.
Note that for the larger loss term, we use a weight smaller than 1 to ensure learning stability.
Fourier Feature Mappings for Learning Periodic System Dynamics
An important feature of ALPS is that it uses a Fourier feature mapping of states to learn periodicsystem dynamics.
Prior works [61, 62] have empirically shown that using a Fourier feature can helpwith high frequency signals.
To lay the foundation for the analysis and justify the use of Fourierfeatures, in Section 5.1 we first review the recent work that uses NTK theory [58, 46].
This allowsus to control the training of our parameter estimator network as fully-connected networks.
Then in
Section 5.2 we use these tools to analyze the effects of using Fourier features, their magnitudes, andtheir phases when predicting system parameters that involve the periodic behavior .
5.1 Deep networks as a kernel regression
Consider a set of labelled training data { ( v i , y i ) } mi with v i ∈ R n , y i ∈ R , and i ∈ [1 : m ] .
Sety =
[ y 1 , . . .
, y m ] T ∈ R m .
Now bring in a feature map ϕ : R n → R r with kernel k ( v i , v j )
=
ϕ ( v i ) T ϕ ( v j ) .
Let K =
[ k ( v i , v j )]
∈ R m × m denote the kernel matrix for the training examplesand k
( v ) =
[ k ( v i , v )]
∈ R m denote the vector of kernel evaluations k ( v i , v ) , i ∈ [1 , m ] , for a testsample v ∈ R n .
The resulting kernel regression predictor is ˆ y ( v ) =
y T K − 1 k ( v ) .
Now bring the concept of NTK proposed by [58].
The theory in [58] says that when the width of thelayers of fully-connected deep networks with weights w initialized from a Gaussian distribution Ntends to infinity, and the learning rate for stochastic gradient descent tends to zero, the neural networkestimator ˆ y ( v ; w ) converges to the kernel regression solution using NTK.
The NTK is defined as
[Equation]
Under asymptotic conditions, a neural network’s output after t updates can be approximated as
[Equation]
where e M is the n by n matrix M given by the power series
e M = (cid:80) ∞
i =
1 i !
M i with M
= I .
Spectral bias in neural networks.
Now we want to compute the training error of a neural networkafter t times update.
Let K
= U Σ U T denote the eigendecomposition of the kernel matrix K whichmust be positive semidefinite (PSD).
Here U is an orthogonal matrix and Σ is a diagonal matrix whoseentries are the nonnegative eigenvalues ordered by magnitude: λ 1 ≥ λ 2 ≥ · · · ≥ λ m ≥ 0 .
So thetraining error in terms of L 2 norm ∥·∥ 22 is
[Equation]
This shows the training convergence will decay exponentially at the rate ηλ i .
Hence the componentsin y will be learned faster if their corresponding eigenvalue is larger.
In Section 5.2 we will showthat for a sequence of states without doing Fourier transform, the resulting NTK will have smallereigenvalues at the high-frequency components.
This leads to a slower convergence in high-frequencycomponents of data which are essential to identify parameters for the periodic behavior.
5.2 The effect of Fourier feature mapping
To understand the effect of the Fourier feature mapping, we first derive the kernel function of Fourierfeatures, their magnitudes, and their phases.
Then we compare the spatial bias of the kernel matricesfrom these three kernels.
Kernels of Fourier feature mapping.
Consider a state trajectory v = [ x 0 , x 1 , . . .
, x τ − 1 ] T , wherehere we consider a 1D case for a scalar state x ∈ R although it can be extended to a vector statex ∈ R n .
The feature maps of the Fourier feature mapping, their magnitudes, and their phases are
[Equation]
where X ω = [Equation] − 1 j =0 x j (cid:104) cos (cid:16) 2 πτ kj (cid:17) − i sin (cid:16) 2 πτ kj (cid:17)(cid:105) .
Set C
k
= (cid:2) cos (cid:0) 2 πτ ki (cid:1) − sin (cid:0) 2 πτ kj (cid:1)(cid:3) ∈
R τ × τ , then the kernel functions of these mappings are
[Equation]
After mapping the input points into the Fourier features, we feed them into a neural network to obtainˆ y ( ϕ ( v ); w ) .
Hence for DFT kernel function the resulting composed kernel of the neural network isk NTK ( ϕ DFT ( v 1 ) , ϕ DFT ( v 2 )) .
Similarly, we have the kernels for k MAG and k PHA .
Visualizing the composed NTK.
We generate time series data composed of different frequencies andmagnitude of sine waves.
The length of the data is 200 samples, and we set τ = 100 ( i.e., v ∈ R 100 )with the sampling rate being 100Hz.
We slide a window and hence have 101 instances in total.
Fig.
3shows the time series data, the effects of each kernel, and its spatial plot.
By construction, k MAG andk PHA have a slower decay in the high-frequency domain as shown in Fig. 3(b).
In addition, k DFTand no mapping have the same kernel matrix and a narrower kernel spatial.
This observation supportsthe idea of not using X ω when learning the system with periodic or vibrational behavior as | X ω |(magnitude) preserves high-frequency information, which is useful for parameter estimations.
Notethat for the system that does not have high-frequency signals, both X ω and | X ω | are equally well.
Comparison to [46].
Our work is inspired by [46], which also uses Fourier feature mappings.
However, there are a few key differences.
(1) Setup.
The Fourier feature mapping in [46] transformsthe low-dimensional x-y coordinates into high-dimensional Fourier features, which projects datainto a high-dimensional space.
In contrast, we use Fourier feature mapping of raw time series data,which compresses data into more compact representations.
(2) Analysis .
We discuss the difference of
Fourier feature mappings, their magnitudes, and their phases to understand the effect of each mappingfor predicting system parameters.
In contrast, [46] does not have such analysis.
Simulations
We study the following three questions: (1) How does ALPS perform compared to other baselinemethods without the Fourier feature mapping and physics-in-the-loop?
(2) What is the effect ofself-attention networks in ALPS?
(3) How does ALPS perform in a real-world time series data?
Visual dataset.
We generate three visual datasets: pendulum, mass-spring-damper (MSD), and atwo-body system to compare the performance of ALPS to that of the baseline in the literature.
Thesebaselines are very commonly used in the prior work.
We first randomly sample an initial state andphysical parameters, and then generate a 125 step rollout following the true system dynamics, andrender corresponding 64 by 64 by 3 pixel observation snapshots.
The sampling rate is 20Hz in thependulum, 100Hz in MSD, and 6Hz in two-body systems.
The observation length τ is 100.
In total,we generate 500 training and 500 test trajectories, resulting in 13,000 training and test sequences.(1)
Pendulum.
The dynamics is ddt
[Equation]
, where φ is an angle, ˙ φ is anangular velocity, β is a friction coefficient, G is a gravitational constant, and L is the length of thependulum.
We fix G = 10 , L = 1 , then sample β from a uniform distribution β ∼ U (0 . 1 , 1) , aninitial angle from a uniform distribution φ 0 ∼ U ( − π, + π ) , and an initial angular velocity from auniform distribution ˙
φ 0 ∼ U (0 . 5 , 4) .
We predict θ =
[ β ] in this task.(2) Mass-spring-damper.
The dynamics of double mass-spring-damper system is
[Equation]
[Table]
[Table]
[Table]
  +   [Table]   (cid:20) u (1) u (2) (cid:21) ,where x (1) , x (3) are the displacement and velocity of the primary spring; x (2) , x (4) are the displacement and velocity of the secondary spring; u (1) , u (2) are the inputs; α (2) , α (4) are the stiffness anddamping ratio of the primary spring and damper; α (1) , α (3) are the stiffness and damping ratio ofthe secondary spring and damper, and m (1) , m (2) are the mass of the primary and secondary spring.
We fix all the parameters and the initial state except for α (2) , which is sampled from a uniformdistribution α (2) ∼ U (4 × 10 3 , 4 × 10 6 ) .
The input excitations are sampled from a unit Gaussiandistribution u ∼ N (0 , 0 . 05)
× N (0 , 0 . 05) ( i.e., white noise).
We predict θ =
[ α (2)] in this task.(3) Two-body.
In this system two particles interact with each other via an attractive force.
Thedynamics is a Hamiltonian H =
∥ p (1) ∥ 22 2 m (1) + ∥ p (2) ∥ 22 2 m (2) + Gm (1) m (2) ∥
q (1) − q (2) ∥ 2 , where p (1) , p (2) are the positionsof the particles; q (1) , q (2) are the momentum of the particles; m (1) , m (2) are the masses of theparticles, and G is a gravitational constant.
We fix G = 10 , m (1) = m (2) = 1 , then sample H froma uniform distribution H ∼ U (0 . 4 , 1) .
We predict θ =
[ H ] in this task.
Time series dataset.
In addition, we obtain an MSD system dataset with state measurements.(4)
Real MSD time series data (wheel suspension system).
To show the applicability of ALPS,we conduct real-world experiments with the data obtained from sensors installed in the train wheelsuspension system by a team of engineers.
We use full-scale dynamics with 18 state elements, 12observations, 16 input excitations, and 24 system parameters.
We want to identify the stiffnessand damping ratio of the 12 spring-dampers.
The dynamics are complex due to the interactions ofmultiple springs and dampers.
In addition, this problem is challenging as the data contain noise.
Thedataset contains 20 trajectories with 500 steps sampled by 100Hz.
We use a 50-50 split to get thetraining and test datasets with τ = 100 .
Here we do not use the self-attention and decoder network in
ALPS–the estimator takes in state measurements directly.
Baselines.
We select the following baselines that are commonly used in the literature.(1) Context-aware dynamics model (CDM) [63].
CDM is the state-of-the-art method to learn thesystem dynamics into two stages: it first learns a context vector that captures the local dynamics, andthen predicts the next state based on the context vector and the current state and input.
CDM doesnot consider physics prior nor Fourier feature mapping.
This is to show that using physics priorsimproves performance and benchmark the results.(2) Autoencoder.
We remove the estimator and the physics simulator to benchmark the mismatchbetween the true state and the latent representation (equal dimension) of the autoencoder in Fig. 1(a)
Pendulum
[Table]
-Spring-Damper
[Table]
Note for
ALPS achieves competitive performance in predicting physical parameters and states. to show the interpretability of ALPS.(3) ALPS w/o the Fourier feature mapping.
We consider a variant by replacing the Fourier featuremapping with raw encoded state trajectories { ˜ x s } .
This method is similar to [45], which also usestime series data to learn system dynamics.(4) ALPS w/o self-attention networks.
We consider a variant by replacing the self-attention networkwith a simple MLP to predict the position of the system, followed by a first-order finite-differenceestimator to estimate the velocity.
This uses the same approach as in [57] to estimate states.
Finally,we ensure that all baselines are comparable in terms of representation power and the number ofparameters.
The loss functions used for these baselines are included in Appendix D.
Evaluation metrics.
We use a mean squared error (MSE) between simulated { ˆ x s } and true states{ x s } to evaluate the performance: SE :
= 1 N (cid:80) Ni =1 (cid:80) ts = t − τ {++
+} ∥
ˆ x s − x s ∥ 22 , where N is the numberof test data.
We also compute MSE between reconstructed { ˆ o t } and true observations { o t } : OE :=1 N (cid:80) Ni
=1 (cid:80) ts = t − τ +
∥
ˆ o s − o s ∥ 22 .
Moreover, we compute the absolute value difference betweenestimated ˆ θ and true parameters θ : PE :
= 1 N (cid:80) Ni [-=
-]=1 (cid:13) (cid:13)(cid:13)
ˆ θ − θ (cid:13) (cid:13)(cid:13) 1 .
Results in the visual tasks.
Fig.
4 shows the results.
We see that (1) ALPS achieves the bestperformance in predicting physical parameters in all cases, with at the most 4.8x lower error inthe pendulum task.
On the other hand, the Fourier feature mapping does not have much effect inpredicting the physical parameter in the two-body system.
This verifies the analysis of the Fourierfeature mapping: for the task with a wider frequency spectrum ( e.g., 100Hz in the MSD), the Fourierfeature improves the prediction due to higher eigenvalues in the high-frequency content, whereas forthe task with only a low-frequency spectrum ( e.g., 6Hz in the two-body), raw state sequences canalready capture low-frequency content.
This supports the idea of using Fourier feature mappings forhigh-frequency data.
(2) ALPS achieves competitive results in predicting the states, with at the most6.3x lower error in the MSD task.
We further visualize the state prediction results in the pudendumtask for ALPS, which shows that ALPS can accurately track the true states.
Without self-attentionnetworks, the network has a substantial SE due to a large error in computing the velocity, as in thetwo-body task.
The smaller the sampling rate is, the greater the velocity estimation error is.
Inaddition, the high SE in autoencoder suggests that its latent representation is uninterpretable.
Thisverifies the idea of using physics to constrain the representation of the autoencoder to estimate states.
In addition, (3) ALPS achieves comparable results in reconstructing observations.
The low error for
CDM in the MSD and two-body tasks is due to degenerated solutions as shown in the Fig.
Thisimplies that using physics stabilizes the training and improves the reconstruction of observations.(4)
Finally, the autoencoder baseline has higher reconstruction loss in some tasks.
This is becausethat it learns a degenerated solution, producing blur images due to high-frequency movements ofobjects on the scene.
Fig.
8 in Appendix E provides more qualitative visualizations of reconstructedimages.
Specifically, we find that CDM fails to track the state and resulting blur images in themass-spring-damper system or duplicated objects in the two-body system.
In contrast, ALPS canprecisely track the state and reconstruct the image well.
Our remaining simulations explore ALPS’sability in predicting physical parameters from raw state measurements. [Figure]
Our method achieves the best performance.
Results in real MSD time series data.
To scalethe proposed approach to a more complex system,we apply the approach in a system with a full-scaletrain wheel suspension system.
The system contains 18 state elements, 12 observations, 16 inputexcitations, and 24 system parameters.
Fig.
5shows the results.
Here ALPS does not use theencoder nor decoder network–the estimator takesin state measurements directly, as the system is fully-observable in the dataset.
Overall we see that(1) ALPS achieves the best performance.
Without using the Fourier feature mapping, the networkcannot learn to identify physical parameters.
(2) CDM has worse SE since an MLP cannot learn wellfrom data with high-frequency content.
In addition, we find that the average prediction error rate( i.e., ˆ θ − θθ ) for these 24 parameters is 0.42 % on the real world complex system.
This also impliesthat by providing physics to the network, we can improve SE.
Moreover, we test ALPS with theinput observations under the same setup.
We found that ALPS is able to achieve an average accuracyof 1.61 %, which is slightly above 0.42 % when using states.
These observations show ALPS canrobustly identify parameters simultaneously from state measurements, and support the theory of using Fourier features to learn periodic dynamics.
Moreover, this result shows ALPS can be deployed inreal prognostic applications for tracking physical parameters to enhance railroad safety.
Our result issignificant for system designers to diagnose the system and hence schedule maintenance.
Discussion of the computational cost. For the training of ALPS, the computation cost mostly comesfrom running the differential solver. For example, under the same batch size of 10 and one gradientupdate, on average ALPS requires 498 seconds whereas the autoencoder requires 317 seconds, whichis 57% of increase. The training time for CDM is also similar to the autoencoder.
The futureimprovement could lie in developing a faster solver.
Conclusion and Future Work
We addressed the problem of predicting states and physical parameters of a system from observationswith dynamic equations.
We showed that the latent representation of the autoencoder is uninterpretable.
We then use dynamic equations to constrain the latent representation of the autoencoderto be consistent with the laws of physics.
We analyzed the effect of Fourier features for estimatingphysical parameters.
The results showed that ALPS achieves competitive performance in the visualtasks and the real raw time series dataset with up to 24 parameter predictions at the same time.
Thisshows that our method scales to high-dimensional systems and it is useful for real-world applications.
Future work could improve ALPS in several ways.
For instance, the understanding of underlyingmechanisms in self-attention networks for estimating states from observations can be advanced.
Inaddition, we require to have dynamic equations and assume that the system is linear and exhibitsperiodic or vibrational behaviors.
One solution is to combine known physics with neural models ( e.g.,interaction networks [13]) to compensate for modeling error, and use time-domain and frequencydomain features together to predict physical parameters.
Moreover, it would be interesting to exploreother more complex tasks ( e.g., non-linear systems, fluid dynamics, molecular simulation).
Onepossible limitation of ALPS is the overhead caused by simulating trajectories.
A future method toovercome this will be valuable.
Finally, incorporating the ideas from the literature on the systemidentification [53, 54, 55] to ALPS will further enhance the performance of ALPS.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We use an autoencoder to estimate states from obser- vations with physics-in-the loop, and show that using the Fourier feature can improve generalization.
Our experiments in Table 4 and Fig.
5 support this idea.
(b) Did you describe the limitations of your work?
[Yes] See Section 7 and Section A in
the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We do
not see any potential negative societal impacts of the work.
We show one example of
applying the proposed model in a real engineering application in Section 6, and discuss the potential impact and limitations in Section A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and Section 5.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Section B and
Section C in the supplementary material.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material
or as a URL)
Section Din the supplementary material.
[-The batch size is 32 ( i.e., 32 rollouts), and we use agradient clipping of 1 . 
-]