Original: G ATED I NFERENCE N ETWORK : I NFERENCING AND L EARNING S TATE -S PACE M ODELS
Revised: G ATED I NFERENCE N ETWORK : I NFERENCING AND L EARNING S TATE -S PACE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: State-space models (SSMs) perform predictions by learning the underlying dynamics of observed sequence.
Revised: State-space models (SSMs) perform predictions by learning the underlying dynamics of observed sequence.

Original: We propose a new SSM approach in both high and low dimensional observation space, which utilizes Bayesian filtering-smoothing to model system’s dynamics more accurately than RNN-based SSMs and can be learned in an end-to-end manner.
Revised: We propose a new SSM approach in both high and low dimensional observation space, which utilizes Bayesian filtering-smoothing to model system’s dynamics more accurately than RNN-based SSMs and can be learned in an end-to-end manner.

Original: The designed architecture, which we call the
Revised: The designed architecture, which we call the

Original: Gated Inference Network (GIN), is able to integrate the uncertainty estimates and learn the complicated dynamics of the system that enables us to perform estimation and imputation tasks in both data presence and absence.
Revised: Gated Inference Network (GIN), is able to integrate the uncertainty estimates and learn the complicated dynamics of the system that enables us to perform estimation and imputation tasks in both data presence and absence.

Original: The proposed model uses the GRU cells into its structure to complete the data flow, while avoids expensive computations and potentially unstable matrix inversions.
Revised: The proposed model uses the GRU cells into its structure to complete the data flow, while avoids expensive computations and potentially unstable matrix inversions.

Original: The GIN is able to deal with any time-series data and gives us a strong robustness to handle the observational noise.
Revised: The GIN is able to deal with any time-series data and gives us a strong robustness to handle the observational noise.

Original: In the numerical experiments, we show that the GIN reduces the uncertainty of estimates and outperforms its counterparts , LSTMs, GRUs and variational approaches.
Revised: In the numerical experiments, we show that the GIN reduces the uncertainty of estimates and outperforms its counterparts , LSTMs, GRUs and variational approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: State estimation and inference in the states in dynamical systems is one of the most interesting problems that has lots of application in signal processing and time series, where the work of Rauch et al.
Revised: State estimation and inference in the states in dynamical systems is one of the most interesting problems that has lots of application in signal processing and time series Rauch et al.

Original: (1965) is among the pioneers of research for learning the dynamic systems.

Original: In some cases, learning state space is a very complicated task due to the relatively high dimension of observations and measurements, which only provides the partial information about the states.
Revised: In some cases, learning state space is a very complicated task due to the relatively high dimension of observations and measurements, which only provides the partial information about the states.

Original: As an instance, consider an image of a pendulum.

Original: The frames reveal the regional information, but they do not give any about the directional information.

Original: Noise is another significant issue in this scenario, where it is more likely to obtain a noisy observation.
Revised: Noise is another significant issue in this scenario, where it is more likely to obtain a noisy observation.

Original: Time series prediction and estimating the next scene, e.g, the state prediction or next observation prediction, is another substantial application that again requires the inference within the states which comes from the observations.
Revised: Time series prediction and estimating the next scene, e.g, the state prediction or next observation prediction, is another substantial application that again requires the inference within the states which comes from the observations.

Original: Classical memory networks such as LSTMs (Hochreiter & Schmidhuber, 1997),GRUs (Cho et al., 2014) and simple
Revised: Classical memory networks such as LSTMs (Hochreiter & Schmidhuber, 1997),

Original:  RNNs like (Wilson & Finkel, 2009) and (Yadaiah & Sowmya, 2006) fail to give some intuition about the uncertainties and dynamics.
Revised: GRUs (Cho et al., 2014) and simple RNNs like (Wilson & Finkel, 2009) and (Yadaiah & Sowmya, 2006) fail to give some intuition about the uncertainties and dynamics.

Original: A bunch of approaches perform the Kalman Filtering (KF) in the latent state which usually requires a deep encoder for feature extraction.
Revised: A group of approaches perform the Kalman Filtering (KF) in the latent state which usually requires a deep encoder for feature extraction.

Original: Krishnan et al.
Revised: Krishnan et al.

Original:  Ghalamzan et al.
Revised: (2017), Ghalamzan et al.

Original: (2021) and Hashempour et al.
Revised: (2021) and Hashempour et al.

Original: (2020) belong to these group of works.
Revised: (2020) belong to these group of works.

Original: However the mentioned solutions have some restrictions, where they are not able to deal with high dimensional non-linear systems and the classic KF approach is computationally expensive, e.g matrix inversion issue.
Revised: However the mentioned solutions have some restrictions, where they are not able to deal with high dimensional non-linear systems and the classic KF approach is computationally expensive, e.g matrix inversion issue.

Original: Likewise, indirect optimization of an objective fuction by using variational inference, like the work of Kingma & Welling (2013), increases the complexity of the model.
Revised: Likewise, indirect optimization of an objective fuction by using variational inference, like the work of Kingma & Welling (2013), increases the complexity of the model.

Original: Moreover, in the variational inference approaches that usually implemented in the context of variational auto encoders for dimension reduction, they do not have access to the loss directly and have to minimize its lower bound instead, which reduce the ability of learning dynamics and affect the performance of the model.
Revised: Moreover, in the variational inference approaches that usually implemented in the context of variational auto encoders for dimension reduction, they do not have access to the loss directly and have to minimize its lower bound instead, which reduce the ability of learning dynamics and affect the performance of the model.

Original: KalmanNet Revach et al.
Revised: KalmanNet Revach et al.

Original: and Ruhe & Forré (2021) use GRU in their structure for the state update.
Revised: and Ruhe & Forré (2021) use GRU in their structure for the state update.

Original: However, they are only able to deal with low-dimensional state space and cannot handle complex high dimensional inputs because of directly using classic Bayesian equations and matrix inversion issue.
Revised: However, they are only able to deal with low-dimensional state space and cannot handle complex high dimensional inputs because of directly using classic Bayesian equations and matrix inversion issue.

Original: Moreover, their structure require the full, or at least partial, dynamic information.
Revised: Moreover, their structure require the full, or at least partial, dynamic information.

Original: The mentioned restrictions for KF and its variants and variational models in addition the necessity of having a metric to measure the uncertainty, motivate us to introduce the GIN, an end to end structure with dynamics learning ability using Bayesian properties for filtering-smoothing.
Revised: The mentioned restrictions for KF and its variants and variational models in addition the necessity of having a metric to measure the uncertainty, motivate us to introduce the GIN, an end to end structure with dynamics learning ability using Bayesian properties for filtering-smoothing.

Original: The 
Revised: The contributions of GIN are: (i) modeling high-low dimensional sequences: we show the eligibility of the GIN to infer both cases by a simple adjustment in the observation transferring functions in the

Original: [Figure]
Revised: [Figure]

Original: contributions of GIN are: (i) modeling high-low dimensional sequences: we show the eligibility of the GIN to infer both cases, where we conduct three experiments of high dimensional non-linear dynamicsobservation for single-double pendulum and visual odometry task and two experiments of low dimensional chaotic observation for Lorenz attractor and real world NCLT dataset. For each case, we compare the GIN with the SOTA approaches like variational EM methods, e.g. Klushyn et al. (2021) and Fraccaro et al. (2017), and Gaussian Process-SSMs (GPSSMs), e.g. Garcia Satorras et al.
Revised: proposed structure, where we conduct three experiments of high dimensional non-linear dynamics and two experiments of low dimensional chaotic observation.

Original: (2019) and Rangapuram et al. (ii) Ability of learning the dynamics(in the lack of them) and utilizing available dynamics(in the presence of them) by introducing GRU cells in the Kalman filtering-smoothing: to  attain more accurate inference of observed dynamical system, we apply
Revised: (ii) Learning/using dynamics: the ability of learning the dynamics(in the lack of them) and utilizing available dynamics(in the presence of them) alongside modeling high-low dimensional observations makes the GIN applicable to a wide range of applications. To attain more accurate inference of observed dynamical system, we apply

Original: GRU cells that increases the modeling capability of the Kalman filtering-smoothing.
Revised: GRU cells that increases the modeling capability of the Kalman filtering-smoothing.

Original: By conduction an ablation study of the GIN being replaced by a linearized Gaussian state transition without nonlinearity, we show the GIN is able for better learning state space representation with disentangled dynamics features.
Revised: By conduction an ablation study of the GIN being replaced by a linearized Gaussian state transition, we show the GIN is able for better learning state space representation with disentangled dynamics features.

Original: (iii) Direct optimization: We show that the posterior and smoothing inference distribution of the state-space model is tractable while dynamics and parameters are estimated by neural networks.
Revised: (iii) Direct optimization: We show that the posterior and smoothing inference distribution of the state-space model is tractable while dynamics and parameters are estimated by neural networks.

Original: Despite variational approaches, this allows us to use recursive Bayesian updates for direct likelihood maximization.
Revised: Despite variational approaches, this allows us to use recursive Bayesian updates for direct likelihood maximization.

Original: (iv) Noise robustness: verified by the numerical results, inferencing for highly distorted sequences is feasible with the GIN.
Revised: (iv) Noise robustness: verified by the numerical results, inferencing for highly distorted sequences is feasible with the GIN.

Original: (v) Missing data imputation: by using 
Revised: (v) Missing data imputation: by using Bayesian properties, the

Original: Bayesian properties, the GIN decides whether to keep the previous information in the memory cell or update them by the obtained observation.
Revised: GIN decides whether to keep the previous information in the memory cell or update them by the obtained observation.

Original: Experimental results show the out-performance of the GIN over the SOTA studies in the imputation task.
Revised: Experimental results show the out-performance of the GIN over the SOTA studies in the imputation task.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: LSTMs and GRUs are RNN structures with the ability of dealing with high-low dimensional data, however, in these models they ignore uncertainty estimation.

Original: To deal with complex sensory inputs, some approaches integrate a deep auto encoders into their architecture.
Revised: To deal with complex sensory inputs, some approaches integrate a deep auto encoders into their architecture.

Original: Among these works, Embed to Control (E2C) (Watter et al., 2015) uses a deep encoder to obtain thelatent observation and a variational inference about the states.
Revised: Among these works, Embed to Control (E2C) (Watter et al., 2015) uses a deep encoder to obtain the observation and a variational inference about the states.

Original: However, these methods are not able to deal with missing data problem and imputation task since they do not rely on memory cells and are not recurrent.
Revised: However, these methods are not able to deal with missing data problem and imputation task since they do not rely on memory cells and are not recurrent.

Original: Another bunch of works like BackpropKF (Haarnoja et al., 2016) and RKN (Becker et al., 2019) apply CNNs for dimension-reduction and output both the uncertainty vector andlatent observation, where they move away from variational inference and borrow Bayesian properties for the inference.
Revised: Another group of works like BackpropKF (Haarnoja et al., 2016) and RKN (Becker et al., 2019) apply CNNs for dimension-reduction and output both the uncertainty vector and observation, where they move away from variational inference and borrow Bayesian properties for the inference.

Original: However, these methods cannot handle the cases with the available knowledge of the dynamics and impose restrictive assumptions over covariance matrices, while the GIN provides a principled way for using the available partial dynamics information and release any assumption over covariance.
Revised: However, these methods cannot handle the cases with the available knowledge of the dynamics and impose restrictive assumptions over covariance matrices, while the GIN provides a principled way for using the available partial dynamics information and release any assumption over covariance.

Original: Toward learning state space (system identification) a bunch of works like Wang et al.
Revised: Toward learning state space (system identification) a group of works like Wang et al.

Original: Ko & Fox (2011) and Frigola et al.
Revised: Ko & Fox (2011) and Frigola et al.

Original: (2013) propose algorithms to learn GPSSMs based on maximum likelihood estimation with the iterative EM algorithm, where filtering-smoothing with a set of fixed parameters γ is conducted (E step), and then updating the set of parameters γ is performed such that the obtained likelihood is maximized (M step).
Revised: (2013) propose algorithms to learn GPSSMs based on maximum likelihood estimation with the iterative EM algorithm.

Original: Frigola et al.
Revised: Frigola et al.

Original: (2013) obtain sample trajectories from the smoothing distribution, then conditioned on this trajectory they conduct M step for the model’s parameters.
Revised: (2013) obtain sample trajectories from the smoothing distribution, then conditioned on this trajectory they conduct M step for the model’s parameters.

Revised: Switching linear dynamics systems (SLDS) (Ghahramani & Hinton, 2000), use additional latent variables to switch among different linear dynamics, where the approximate inference algorithms can be utilized to model switching linearity for reducing approximation errors ,however, this approach is not as flexible as general non-linear dynamical systems because the switch transition model is assumed independent of states and observations.

Revised: To address this problem, Linderman et al.

Revised: (2017) performs SLDS method through augmentation with a Polya-gamma-distributed variable and a stick-breaking process, however, this approach employs Gibbs sampling for inferring the parameters and therefore is not scalable to large datasets.

Revised: Auto-regressive Hidden Markov Models (ARHMM) explain time series structures by defining a mapping from past observations to the current observation.

Revised: (Salinas et al., 2020) is a

Original:  Other group of works consider EM-based variational-inference likeStructured Inference Networks (SIN)
Revised: ARHMM approach, in which target values are used as inputs directly. However, this dependency of the model on the targets makes the model more vulnerable to noise. This issue is addressed in DSSM (Rangapuram et al., 2018), another ARHMM approach, where the target values are only incorporated through the likelihood term. Other group of works consider EM-based variational-inference like

Original:  (Krishnan et al., 2017), where it utilizes a RNN to update the state.
Revised: Structured Inference Networks (SIN) (Krishnan et al., 2017), where it utilizes a RNN to update the state.

Original: Kalman Variational Autoencoder (KVAE)
Revised: Kalman Variational Autoencoder (KVAE)

Original: (Fraccaro et al., 2017) and Extended KVAE (EKVAE) (Klushyn et al., 2021) use the original KF equations and apply both filtering and smoothing.
Revised: (Fraccaro et al., 2017) and Extended KVAE (EKVAE) (Klushyn et al., 2021) use the original KF equations and apply both filtering and smoothing.

Original: However, these EM-based variational inference methods are not able to estimate the states directly because of optimizing the lower bound of likelihood.
Revised: However, these EM-based variational inference methods are not able to estimate the states directly because of optimizing the lower bound of likelihood.

Original: Extra complexity is another issue with these approaches, while they are addressed by the proposed structure and direct end-to-end optimization in the GIN.
Revised: Extra complexity is another issue with these approaches, while they are addressed by the proposed structure and direct end-to-end optimization in the GIN.

Original: We compare the GIN with these approaches in the experiment section and provide an empirical complexity analysis in appendix A.8.2.
Revised: We compare the GIN with these approaches in the experiment section and provide an empirical complexity analysis in appendix A.8.2.

Original: We provide a detailed discussion of recent related work in appendix A.8.3.
Revised: We provide a detailed discussion of recent related work in appendix A.8.3.

Original:  G ATED I NFERENCE N ETWORK FOR S YSTEM I DENTIFICATION
Revised:  G ATED I NFERENCE N ETWORK FOR S YSTEM I DENTIFICATION

Original: In the context of System Identification (SI), i.e. when we lack the dynamics, the GIN is similar to a Hammerstein-Wiener (HW) model (Schoukens & Tiels, 2017)
Revised: In the context of System Identification (SI) the GIN is similar to a Hammerstein-Wiener (HW) model (Schoukens & Tiels, 2017)

Original: (Gilabert et al., 2005), in the sense that it estimates the system parameters directly from the observations, which is in the figure 2.
Revised: (Gilabert et al., 2005), in the sense that it estimates the system parameters directly from the observations, which is in the figure 2.

Original: e ( . ) and d ( . ) are implemented with non-linear functions, e.g. auto encoders-MLPs.
Revised: e ( . ) and d ( . ) are implemented with non-linear functions, e.g. auto encoders-MLPs.

Original: Transition block in figure 2 represents the dynamics of the system that allows for the inference using the Gaussian state space filtering-smoothing equations.
Revised: Transition block in figure 2 represents the dynamics of the system that allows for the inference using the Gaussian state space filtering-smoothing equations.

Original: However unlike a HW model, we employ non-linear GRU cells in the transition block that calculate the Kalman Gain (KG) and smoothing gain (SG) in an appropriate manner by circumventing the computational complexity, i.e matrix inversion issues.
Revised: However unlike a HW model, we employ non-linear GRU cells in the transition block that calculate the Kalman Gain (KG) and smoothing gain (SG) in an appropriate manner by circumventing the computational complexity, i.e matrix inversion issues.

Original: GRU cells empower the whole system by applying non-linearity to the linearized Gaussian state space models (LGSSMs).
Revised: GRU cells empower the whole system by applying non-linearity to the linearized Gaussian state space models (LGSSMs).

Original: Numerical results indicate that by the proposed structure, having a good inference for even the complex non-linear systems with high dimensional observations is feasible.
Revised: Numerical results indicate that by the proposed structure, having a good inference for even the complex non-linear systems with high dimensional observations is feasible.

Original: To achieve this, we assume the state fits into Gaussian state space models (GSSMs), which are commonly used to model sequences of vectors.
Revised: To achieve this, we assume the state fits into Gaussian state space models (GSSMs), which are commonly used to model sequences of vectors.

Original: In most cases, the dynamics of the system might not be available or hard to obtain; while the process noise and observation noise are unknown (our first three experiments).
Revised: In most cases, the dynamics of the system might not be available or hard to obtain; while the process noise and observation noise are unknown (our first three experiments).

Original: Accordingly, we construct GIN to learn unknown variables from data in an end to end fashion, then we utilize the constructed KG and SG during inference time to obtain the filteredsmoothed states.
Revised: Accordingly, we construct GIN to learn unknown variables from data in an end to end fashion, then we utilize the constructed KG and SG during inference time to obtain the filteredsmoothed states.

Original: The proposed architecture is depicted in figure 3.
Revised: The proposed architecture is depicted in figure 4.

Original: In the presence of dynamics (our last two experiments), autoencoder and Dynamics Network in figureare removed. The architecture for operating high and low dimensional observations are depicted separately with further details in appendix figures 8 and 9.
Revised: In the presence of dynamics (our last two experiments), auto-encoder and Dynamics Network in figure 4 are replaced by MLP to model the observation noise.

Original:  P ARAMETERIZATION
Revised:  P ARAMETERIZATION

Original: In this section we show the parameterization of the inference model.
Revised: In this section we show the parameterization of the inference model.

Original: Given original noisy measurements o 1: T and latent noisy observations  w 1: T , we want to find good estimate of  thelatent states x 1: T  .
Revised: Firstly, we refer the readers to the summary of Kalman filtering-smoothing background for the completeness in appendix A.2. In the rest of the paper we use the following notations, where the original observations are o 1: T , the transferred observations are w 1: T and R 1: T are diagonal matrices with r 1: T diagonal elements that correspond to the covariance of transferred observation noise. x 1: T corresponds to the states and Q 1: T are diagonal matrices with q 1: T diagonal elements which are the covariance of state process noise. ( x − t , Σ − t ) are the mean vector and covariance matrix of the prior state at time step t , i.e. p ( x t | w 1: t − 1 ) , and ( x + t , Σ + t ) are the mean vector and covariance matrix of the posterior state at time step t , i.e. p ( x t | w 1: t ) .

Revised: We define the transition matrices F 1: T and emission matrices H 1: T as the dynamics of the model, where lack of dynamics in the first three experiments means that ( F 1: T , H 1: T ) are not know and are going to be trained(graphical model is in figure 3), while the presence of dynamics in our last two experiments means that ( F 1: T , H 1: T ) are known(graphical models are in figures 9a and 9b).

Original: To achieve this, we want to infer the marginal distributions p ( x  t | w 1: t ) for the online inference approach or filtering; and p ( x  t| w 1: T ) for the full inference approach or smoothing.
Revised: The dynamics ( F 1: T , H 1: T ) and noise matrices In the filtering step, by using the priors ( x − t , Σ − t ) and the observation ( w t , r t ) , the next posteriors ( x + t , Σ + t ) are obtained. Applying smoothing operation over the obtained posteriors ( x + t , Σ + t ) is feasible in the smoothing step. Finally, the decoder(MLP) is utilized to produce o + t , which can be the high-low dimensional noise free estimates.

Original: By defining γ t = ( F t , H t , Q t , R t ) as the model parameters at time t , where ( Q t , R t ) are process and observation noise and  ( F t , H t  ) explain the state transition and observation emission, we introduce an advantageous prediction parameterization as p γ t ( x t | x t − 1 , w 1: t − 1 ) =
Revised: ( R 1: T , Q 1: T ) construct the system parameters γ 1: T = ( F 1: T , H 1: T , Q 1: T , R 1: T ) . Given original observations o 1: T and transferred observations w 1: T , we want to find good estimate of the latent states x 1: T . To achieve this, we want to infer the marginal distributions p ( x t | w 1: t ) for the online inference approach or filtering; and p ( x t | w 1: T ) for the full inference approach or smoothing. We introduce an advantageous prediction parameterization as p γ t ( x t | x t − 1 , w 1: t − 1 ) =

Original: N ( F t x t − 1 , Q t ) , where x t − 1 ∼  N ( x + t − 1 , Σ + t − 1 ) .
Revised: N ( F t x t − 1 , Q t ) , where x t − 1 ∼ p γ t − 1 ( x t − 1 | w 1: t − 1 ) = N ( x + t − 1 , Σ + t − 1 ) .

Original: Then, p γ t ( x t | w 1: t − 1 ) =
Revised: Then, p γ t ( x t | w 1: t − 1 ) =

Original: N ( F t x + t − 1 , F t Σ + t − 1 F Tt + Q t ) =
Revised: N ( F t x + t − 1 , F t Σ + t − 1 F Tt + Q t ) =

Original: N ( x − t , Σ − t ) is obtained by marginalizing out x t − 1 and the Gaussianity of p γ t ( x t | w 1: t − 1 ) results from the Gaussianity of prediction parameterization.
Revised: N ( x − t , Σ − t ) is obtained by marginalizing out x t − 1 and the Gaussianity of p γ t ( x t | w 1: t − 1 ) results from the Gaussianity of prediction parameterization.

Original: By having p γ t ( x t | w 1: t − 1 ) and observing w t , filtering parameterization is introduced as:
Revised: By having p γ t ( x t | w 1: t − 1 ) and observing w t , filtering parameterization is introduced as:

Original: [Equation]
Revised: [Equation]

Original: where K t is KG.
Revised: where K t is KG.

Original: After observing all latent observations w 1: T , one can do backward induction and propagate to the previous states using the chain rule.
Revised: After observing all transferred observations w 1: T , one can do backward induction and propagate to the previous states using the chain rule.

Original: This procedure, known as smoothing, can be parameterized as:
Revised: This procedure, known as smoothing, can be parameterized as:

Original: [Equation]
Revised: [Equation]

Original: where J t is SG and we use short handed notation N ( x t | T , Σ t | T ) instead of (2).
Revised: where J t is SG and we use short handed notation N ( x t | T , Σ t | T ) instead of (2) .

Original: Full derivation of filtering-smoothing parameterization is in appendex A.2.

Original: These parameterizations give us some insight to 1-illustrate a tractable way to construct p γ ( x | w ) and accordingly obtain the posterior and smoothened states, based on which o
Revised: These parameterizations give us some insight to 1-illustrate a tractable way to construct p γ ( x | w ) and accordingly obtain the posterior and smoothened states, based on which o

Original: + is constructed and 2- appropriately modeling γ and KG(SG) with neural networks.
Revised: + is constructed and 2- appropriately modeling γ and KG(SG) with neural networks.

Original: To construct the KG and SG networks, we have to find appropriate inputs containing related information to attain the KG and SG.
Revised: To construct the KG and SG networks, we have to find appropriate inputs containing related information to attain the KG and SG.

Original: In (1) and (2), KG and SG are given by (3) and (4), respectively.
Revised: In (1) and (2), KG and SG are given by (3) and (4), respectively.

Original: [Equation]
Revised: [Equation]

Original: ∝ Σ [Equation] 4) (3) is proportional to the prior covariance at time t , Σ − t , and the observation noise matrix, R t , while (4) is proportional to prior covariance matrix at time t+1 , Σ − t
Revised: (3) is proportional to the prior covariance at time t , Σ − t , and the observation noise matrix, R t , while (4) is proportional to prior covariance matrix at time t

Original: +1  .
Revised: +1 , Σ − t +1 .

Original: Our encoder(MLP) directly maps the observation noise matrix from the observation space, but the state covariance is a recursive function of previous states.
Revised: Our encoder(MLP) directly maps the observation noise matrix from the observation space, but the state covariance is a recursive function of previous states.

Original: Consequently, we consider GRU KG and GRU SG which are networks including
Revised: Consequently, we consider GRU KG and GRU SG which are networks including

Original: GRU that map [ f ( Σ − t ) , R t ] and f ( Σ − t + 1 ) to the KG and SG, respectively.
Revised: GRU that map [ f ( Σ − t ) , R t ] and f ( Σ − t + 1 ) to the KG and SG, respectively.

Original: GRU KG considers R t , a diagonal matrix with r t elements in figure 3, as a part of its input to incorporate the effects of observation noise.
Revised: GRU KG considers R t , a diagonal matrix with r t elements in figure 4, as a part of its input to incorporate the effects of observation noise.

Original: In the case of high dimensional state space, due to the high dimension of Σ − t and Σ − t + 1 , f is a convolutional layer with pooling to extract the valuable information of the covariance matrix that reduces its size, while for the low dimension of Σ − t and Σ − t + 1 ,  the identity function is used for f .
Revised: In the case of high dimensional state space, due to the high dimension of Σ − t and Σ − t + 1 , f is a convolutional layer with pooling to extract the valuable information of the covariance matrix that reduces its size, while for the low dimension of Σ − t and Σ − t + 1 , f is the identity function.

Original: Such structure does not have extreme negative effects on the performance since in the reality covariance matrix is sparse and we can code the information into smaller dimension, without loosing the general information.

Original: Inference in the latent state space can now be implemented by learning the process noise, prediction step, filtering step, smoothing step and learning dynamics.

Original: Learning The Process Noise.
Revised: Learning The Process Noise.

Original: In the filtering procedure, the process noise in time t is obtained as
Revised: In the filtering procedure, the process noise in time t is obtained as

Original: [Equation]
Revised: [Equation]

Original: (5) where Σ − t , F t and Σ + t − 1 are prior state covariance, transition matrix and posterior state covariance at time t .
Revised: where Σ − t , F t and Σ + t − 1 are prior state covariance, transition matrix and posterior state covariance at time t .

Original: It is shown that Q t can be written as a function of F t , while the derivations are rather lengthy, therefore, we refer to the appendix materials A.3.
Revised: It is shown that Q t can be written as a function of F t as [Equation] while the derivations are rather lengthy, therefore, we refer to the appendix materials A.3.

Original: From (31), the relation of the process noise with the transition matrix indicates that F t can possess the effects of Q t if we learn it in an appropriate manner.
Revised: From (32), the relation of the process noise with the transition matrix indicates that F t can possess the effects of Q t if we learn it in an appropriate manner.

Original: ˆ F t ( Q t ) notation means that the learned transition matrix ˆ F t comprises the effects of Q t , while for the simplicity we use ˆ F t abbreviation.
Revised: ˆ F t ( Q t ) notation means that the learned transition matrix ˆ F t comprises the effects of Q t , while for the simplicity we use ˆ F t abbreviation.

Original: Therefore, it is possible to rewrite (5) as
Revised: Therefore, it is possible to rewrite (5) as

Original: [Equation]
Revised: [Equation]

Original: [Equation] Another way to have a a meaningful inference about the process noise matrix is to obtain it from (29) as a recursive function of x + t − 1 and Q t − 1 .
Revised: Another way to have a a meaningful inference about the process noise matrix is to obtain it from (30) as a recursive function of x + t − 1 and Q t − 1 .

Original: Intuitively, g function in (29) that we call it Q Network , can be implemented by a memory cell, e.g., a GRU cell, to keep the past status of Q ,however, it increases the complexity of the model.
Revised: Intuitively, g function in (30) that we call it Q Network , can be implemented by a memory cell, e.g., a GRU cell, to keep the past status of Q ,however, it increases the complexity of the model.

Original: Equivalently, one can obtain Q t directly from x + t − 1  as stated in (31), where a fully connected with a positive activation function plays the rule of g that maps x + t −to Q t . Numerical results corresponding to the mentioned methods for inference in the process noise can be found in A.10.
Revised: Equivalently, one can obtain Q t directly from x + t − 1 with MLP as stated in (32).

Original: Both of these solutions can be utilized when the dynamics are known, i.e. we cannot learn the effects of Q t jointly with F t as F t is not trainable.
Revised: Both of these solutions can be utilized when the dynamics are known, i.e. we cannot learn the effects of Q t jointly with F t as F t is not trainable.

Original: Prediction Step.
Revised: Prediction Step.

Original: Similar to the model based Kalman Filter, by using dynamics of the system and transition, the next priors are obtained from the current posterior by
Revised: Similar to the model based Kalman Filter, by using dynamics of the system and transition, the next priors are obtained from the current posterior by

Original: [Equation]
Revised: [Equation]

Original: where ˆ F t is the learned transition matrix comprises the effects of the process noise from previous section.
Revised: where ˆ F t is the learned transition matrix comprises the effects of the process noise from previous section.

Original: By which, it is feasible to predict state mean and the state covariance matrix.
Revised: By which, it is feasible to predict state mean and the state covariance matrix.

Original: Filtering Step.
Revised: Filtering Step.

Original: To obtain the next posteriors based on the new observation ( w t , r t ) , i.e. the output of e ( . ) in figure 2, we have to use the obtained KG matrix from GRU KG network and learned emission matrix ˆ H t to complete updating the state mean vector and state covariance matrix.
Revised: To obtain the next posteriors based on the new observation ( w t , r t ) , i.e. the output of e ( . ) in figure 2, we have to use the obtained KG matrix from GRU KG network and learned emission matrix ˆ H t to complete updating the state mean vector and state covariance matrix.

Original: This procedure is given by
Revised: This procedure is given by

Original: [Equation]
Revised: [Equation]

Original: In addition to avoiding the matrix inversion that arises in the computation of Kalman gain and applying non-linearity to handle more complex dynamics, the architecture of KG network, GRU KG , can reduce the dimension of the input to its corresponding GRU cell, and thus reduces the total amount of parameters quadratically.
Revised: In addition to avoiding the matrix inversion that arises in the computation of Kalman gain and applying non-linearity to handle more complex dynamics, the architecture of KG network, GRU KG , can reduce the dimension of the input to its corresponding GRU cell, and thus reduces the total amount of parameters quadratically.

Original: Additionally, positive r t vector and Cholesky factor consideration, M t M Tt in (8), makes this procedure trivial to guarantee the positive definiteness of the resulted covariance matrices.
Revised: Additionally, positive r t vector and Cholesky factor consideration, M t M Tt in (9), guarantee the positive definiteness of the resulted covariance matrices.

Original: Smoothing Step.
Revised: Smoothing Step.

Original: After obtaining filtered states ( x +1: T , Σ +1: T ) in filtering step, we employ smoothing properties of Bayesian to get smoothed version of the states.
Revised: After obtaining filtered states ( x +1: T , Σ +1: T ) in filtering step, we employ smoothing properties of Bayesian to get smoothed version of the states.

Original: In this stage, we use J 1: T matrices obtained from GRU SG network, learned transition matrices ˆ F 1: T and filtered states ( x +1: T , Σ
Revised: In this stage, we use J 1: T matrices obtained from GRU SG network, learned transition matrices ˆ F 1: T and filtered states ( x +1: T , Σ

Original: +1: T ) .
Revised: +1: T ) .

Original: The procedure in each smoothing step is given by:
Revised: The procedure in each smoothing step is given by:

Original: [Equation]
Revised: [Equation]

Original:  where the first smoothing state is set to the last filtering state, i.e. ( x T | T , Σ T | T ) 
Revised: 12) where the first smoothing state is set to the last filtering state, i.e. ( x T | T , Σ T | T ) =

Original: = ( x + T , Σ + T ) .
Revised: ( x + T , Σ + T ) .

Original: Learning Dynamics.
Revised: Learning Dynamics.

Original: We can model the dynamics in each time step t as a function of the latent observations w 1: t − 1 .
Revised: We can model the dynamics in each time step t as a function of the transferred observations w 1: t − 1 .

Original: However, conditioning on the noisy observations can distort the procedure of learning the dynamics.
Revised: However, conditioning on the noisy observations can distort the procedure of learning the dynamics.

Original: Instead, we use the state x + t − 1 in GSSM that includes the history of the system with considerable lower noise distortion to increase system’s noise robustness, where we generate time correlated noise in our experiments to show this robustness(see appendix A.5).
Revised: Instead, we use the state x + t − 1 in GSSM that includes the history of the system with considerable lower noise distortion to increase system’s noise robustness, where we generate time correlated noise in our experiments to show this robustness(see appendix A.5).

Original: In other words, original transition and emission equations, x t = f
Revised: In other words, original transition and emission equations, x t = f

Original: ( x t − 1 ) + q t and w t
Revised: ( x t − 1 ) + q t and w t

Original: = h ( x t ) + r t , are modeled as x t = ˆ F t ( x + t − 1 ) x
Revised: = h ( x t ) + r t , are modeled as x t = ˆ F t ( x + t − 1 ) x

Original: t − 1 + q t and w t =
Revised: t − 1 + q t and w t =

Original: ˆ H t ( x + t − 1 ) x t + r t .
Revised: ˆ H t ( x + t − 1 ) x t + r t .

Original: We learn K state transition and emission matrices ˆ F k and ˆ H k , and combine each one with the state dependent coefficient α k ( x + t − 1 ).
Revised: We learn K state transition and emission matrices ˆ F k and ˆ H k , and combine each one with the state dependent coefficient α k ( x + t − 1 ).

Original: [Equation]
Revised: [Equation]

Original: where a separated neural network with softmax output is utilized to learn α k ( x + t − 1 ) that we call it
Revised: where a separated neural network with softmax output is utilized to learn α k ( x + t − 1 ) that we call it

Original: Dynamics Network .
Revised: Dynamics Network .

Original: This formulation enables us to follow Bayesian methodology.
Revised: This formulation enables us to follow Bayesian methodology.

Original: Despite classic
Revised: Despite classic

Original: LGSSMs that are not able to learn the dynamics, e.g. EKF and UKF, the trainable dynamics in the GIN are function of the states.
Revised: LGSSMs that are not able to learn the dynamics, e.g. EKF and UKF, the trainable dynamics in the GIN are function of the states.

Original: For the notation simplicity, we have used (ˆ F t , ˆ H t ) instead of (ˆ F t ( x + t − 1 ) , ˆ H t ( x + t − 1 )) in the paper.
Revised: For the notation simplicity, we have used (ˆ F t , ˆ H t ) instead of (ˆ F t ( x + t − 1 ) , ˆ H t ( x + t − 1 )) in the paper.

Original: Intuitively, each of k sets of ˆ F k , ˆ H k models different dynamics, that will dominate when the corresponding element of the dynamics network is high.

Original: Additionally, this formulation help us to introduce a loss term which tries to increase the distance of each pair of ˆ F k , ˆ H k set.

Original: We found this as a potential solution to prevent the model goes through the mode collapse and prevent converging to poor local minima during training dynamics(see appendix A.8).

Original:  F ITTING
Revised:  F ITTING

Original: For the state estimation task, by implementing p ( w 1: T | o 1: T ) , p ( x t | w 1: T ) and p ( s t | x t ) with encoder, smoothing parameterization and decoder, we maximise the log-likelihood of output p ( s t | o 1: T ) =
Revised: For the state estimation task, by implementing p ( w 1: T | o 1: T ) , p ( x t | w 1: T ) and p ( s t | x t ) with encoder, smoothing parameterization and decoder, we maximise the log-likelihood of output p ( s t | o 1: T ) =

Original: (cid:82) p [Equation] T [Equation]
Revised: (cid:82) p ( s t | x t ) p ( x t | w 1: T ) p ( w 1: T | o 1: T )

Original:  w [Equation] T ,where s t is the estimated state, i.e. equal to o + t in figure 3.
Revised: d x t d w 1: T ,where s t is the estimated state, i.e. equal to o + t in figure 4.

Original: For the image imputation task, in addition to the state likelihood, we add the reconstruction pseudo-likelihood for inferring images by using Bernoulli distributions as p ( i t | x t ) , i.e. the decoder in figure 3 maps both state s t and image i t : o + t =
Revised: For the image imputation task, in addition to the state likelihood, we add the reconstruction pseudo-likelihood for inferring images by using Bernoulli distributions as p ( i t | x t ) , i.e. the decoder in figure 4 maps both state s t and image i t : o + t =

Original: [ i t , s t ] .
Revised: [ i t , s t ] .

Original: Further details of distribution assumptions and hyper parameter optimization can be found in the appendix A.4 and A.8.
Revised: Further details of distribution assumptions and hyper parameter optimization can be found in the appendix A.4 and A.8.

Original: After training phase, forecasting desired number of time steps is applicable by plugging the new value x t = ˆ F t x t −recursively in the model, and so on.
Revised: After training phase, forecasting desired number of time steps is applicable by plugging the new value x t = ˆ F t x t −recursively in the model, and so on.

Original: Filtering procedure, which includes Dynamics Network , prediction step and filtering step, can be considered as a memory network.

Original: Similarly, smoothing procedure that includes smoothing step is another memory network so that we can calculated the gradients by using (truncated) BPTT Werbos (1990) to train the GRU cells inside the architecture in an end to end manner.

Original: To prevent GRU cells from potential instability, we split lengthy sequences into smaller pieces( ≤ 200) and pass them separately.

Original: Likelihood for Inferring States.
Revised: Inferring States.

Original: Consider the ground truth sequence is defined as s 1 : T .
Revised: Consider the ground truth sequence is defined as s 1 : T .

Original: We determine the log likelihood of the states as:
Revised: We determine the log likelihood of the states as:

Original: [Equation]
Revised: [Equation]

Original: where the dec mean ( . ) and dec covar ( . ) determines those parts of the decoder that are used to obtain the state mean and state variance, respectively.
Revised: where the dec mean ( . ) and dec covar ( . ) determines those parts of the decoder that are used to obtain the state mean and state variance, respectively.

Revised: We use Wishart distribution as a prior for our estimated covariance matrix, which pushes the estimated covariance toward a scale of identity matrix and the scale is a hyper parameter.

Revised: Such prior prevents getting high log-likelihood due to the high uncertainty.

Original: Likelihood for inferring images.
Revised: Likelihood for inferring images.

Original: For the imputation task, consider the ground truth as the sequence of images and their corresponding states, which are defined as [ s 1: T , i 1: T ] and the dimension of i t is
Revised: For the imputation task, consider the ground truth as the sequence of images and their corresponding states, which are defined as [ s 1: T , i 1: T ] and the dimension of i t is

Original: D o .
Revised: D o .

Original: We determine the log likelihood:
Revised: We determine the log likelihood:

Original: [Equation]
Revised: [Equation]

Original:  dec k ( x t ) defines the corresponding part of the decoder that maps the k -th pixel of i t image and λ constant determines the importance of the reconstruction.
Revised: (15) dec k ( x t ) defines the corresponding part of the decoder that maps the k -th pixel of i t image and λ constant determines the importance of the reconstruction.

Original: The first term in RHS is obtained from (13) and we abbreviate the second term as L ( i 1: T ) .
Revised: The first term in RHS is obtained from (14) and we abbreviate the second term as L ( i 1: T ) .

Original:  E VALUATION AND E XPERIMENTS
Revised:  E VALUATION AND E XPERIMENTS

Original: We divide our experiments into two parts, first the tasks in which the observation space is high dimensional like sequence of images, and second the applications that the observation is in low dimension by itself so there is no need to include encoder for dimension reduction.
Revised: We divide our experiments into two parts, first the tasks in which the observation space is high dimensional like sequence of images, and second the applications that the observation is in low dimension by itself so there is no need to include encoder for dimension reduction.

Original: The training algorithms of both cases are added in the appendix section A.11. 
Revised: The training algorithms of both cases are added in the appendix section A.11. [Table] [Table] [Table]

Revised: [Table]

Original: 6. H IGH D IMENSIONAL O BSERVATION WITH L ACK OF D YNAMICS
Revised: 6. H IGH D IMENSIONAL O BSERVATION WITH L ACK OF D YNAMICS

Original: We include three high dimensional experiments.
Revised: We include three high dimensional experiments.

Original: The first two experiments are single pendulum and double pendulum, where the dynamics of the latter one is more complicated and also another difficulty exists that the second link of the double pendulum may be occluded by the first link during the simulation.
Revised: The first two experiments are single pendulum and double pendulum, where the dynamics of the latter one is more complicated.

Original: The last experiment is visual odometry task.
Revised: The last experiment is visual odometry task.

Original: Intuitive python code in appendix A.12.1.
Revised: Intuitive python code in appendix A.12.1.

Original: [Table]

Original: [Table]

Original: [Table]

Original: consider n = 3 m , intuitively the last part of the state is dedicated to the acceleration information causing a more lieklihood.

Original: [Table]

Original: 6.1. S INGLE P ENDULUM AND D OUBLE P ENDULUM
Revised: 6.1. S INGLE P ENDULUM AND D OUBLE P ENDULUM

Original: The inputs of the encoder are the images with size of 24 × 24 .
Revised: The inputs of the encoder are the images with size of 24 × 24 .

Original: The angular velocity is disturbed by transition noise which follows Normal distribution with σ = 0 .
Revised: The angular velocity is disturbed by transition noise which follows Normal distribution with σ = 0 .

Original: 1 as its standard deviation at each step.
Revised: 1 as its standard deviation at each step.

Original: In the pendulum experiment [Table] we perform the filtering-smoothing by the GIN where the observation is distorted with high observation noise.
Revised: In the pendulum experiment, we perform the filtering-smoothing by the GIN where the observation is distorted with high observation noise.

Original: Furthermore, we compare GIN with LGSSM, where the
Revised: Furthermore, we compare GIN with LGSSM, where the

Original: GRU cells are omitted from the GIN structure and classic filtering-smoothing equations are used, instead.
Revised: GRU cells are omitted from the GIN structure and classic filtering-smoothing equations are used, instead.

Original: The distortion with noise changes between noiseless situation to the whole image distorted with noise such that pure noise is observed.

Original: Moreover, the cell may observe fully distorted image for consecutive time steps, which means that the noise has correlation with time.

Original: The log-likelihood and squared error (SE) of positions for single and double pendulum are given in table 2 and 1, respectively.
Revised: The log-likelihood and squared error (SE) of positions for single and double pendulum are given in table 2 and 1, respectively.

Original: Generated samples from trained smooth-filter distributions are in appendix figures 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 and 31.
Revised: Generated samples from trained smooth-filter distributions are in appendix figures 16-33.

Original: By randomly deleting the half of images from the generated sequences, we conduct the image imputation task to our model by predicting those missing parts, while the missingness applied to train and test are not same, but random.
Revised: By randomly deleting the half of images from the generated sequences [Table] we conduct the image imputation task to our model by predicting those missing parts, while the missingness applied to train and test are not same, but random.

Original: The results are in table 3 and 4.The GIN outperforms all the other models, although the variational inference models have more complex structures in KAVEand EKVAE.
Revised: The results are in table 3 and 4.The GIN outperforms all the other models, although the variational inference models have more complex structures in KAVE

Original: We include the results using the MSE as well, to illustrate that our approach is also competitive in prediction accuracy (See A.10).

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: , beginning from up to down, indicates the ground truth, uninformed observation and the imputation results of the GIN(smoothed). Missingness is applied randomly for train and test.

Original: [Table]
Revised: [Table]

Original: Each figure, beginning from up to down, indicates the ground truth, uninformed observation and the imputation results of the GIN(smoothed).
Revised: Each figure, beginning from up to down, indicates the ground truth, uninformed observation and the imputation results of the GIN(smoothed) and EKVAE.

Revised: We include the results using the MSE as well, to illustrate that our approach is also competitive in prediction accuracy (See A.10).

Original: 6.1. V ISUAL O DOMETRY OF KITTI D ATASET
Revised: 6.1. V ISUAL O DOMETRY OF KITTI D ATASET

Original: We also evaluate the GIN with the higher dimensional observations for the visual odometry task on the KITTI dataset Geiger et al.
Revised: We also evaluate the GIN with the higher dimensional observations for the visual odometry task on the KITTI dataset Geiger et al.

Original: This dataset consists of 11 separated image sequences with their corresponding labels.
Revised: This dataset consists of 11 separated image sequences with their corresponding labels.

Original: In order to extract the positional features, we use a feature extractor network proposed by Zhou et al.
Revised: In order to extract the positional features, we use a feature extractor network proposed by Zhou et al.

Original: in Zhou et al.
Revised: in Zhou et al.

Original: The obtained features are considered as the observations of the GIN, i.e. ( w , r ).
Revised: The obtained features are considered as the observations of the GIN, i.e. ( w , r ).

Original: Additionally, we compare the results with LSTM, GRU, 
Revised: Additionally, we compare the results with LSTM, GRU, DeepVO

Original: DeepVO Wang et al.
Revised: Wang et al.

Original: (2017) and KVAE.
Revised: (2017) and KVAE.

Original: The results are in table 5 and figure 7, where the common evaluation scheme for the KITTI dataset is exploited.
Revised: The results are in table 5 and figure 8, where the common evaluation scheme for the KITTI dataset is exploited.

Original: The results of the KVAE degrades substantially as we have to reduce the size of the latent observation to prevent the complexity of matrix inversion in the smoothing-filtering, which causes an inevitable information loss.
Revised: The results of the KVAE degrades substantially as we have to reduce the size of the transferred observation to prevent the complexity of matrix inversion.

Original: 6. L OW D IMENSIONAL O BSERVATION WITH P RESENCE OF D YNAMICS
Revised: 6. L OW D IMENSIONAL O BSERVATION WITH P RESENCE OF D YNAMICS

Original: We conduct two experiments for this case, Lorenz attractor problem and the real world dynamics NCLT dataset, where we are aware of the dynamics.
Revised: We conduct two experiments, Lorenz attractor problem and the real world dynamics NCLT dataset, where we are aware of the dynamics.

Original: Intuitive python code in appendix A.12.2.
Revised: Intuitive python code in appendix A.12.2.

Revised: However, the GIN is able to deal the cases in which the dynamics are not known.

Revised: To show this, we conduct additional experiment in the appendix, where we do not give the dynamics information of Lorenz attractor and

Revised: NCLT dataset to the model, so that they will be learned(see figures 42-46).

Original: 6.2. L ORENZ A TRRACTOR
Revised: 6.2. L ORENZ A TRRACTOR

Original: The Lorenz system is a system of ordinary differential equations that describes a non-linear dynamic system used for atmospheric convection.
Revised: The Lorenz system is a system of ordinary differential equations that describes a non-linear dynamic system used for atmospheric convection.

Original: Due to nonlinear dynamics of this chaotic system (see A.6), it can be a good evaluation for the GIN cell.
Revised: Due to nonlinear dynamics of this chaotic system (see A.6), it can be a good evaluation for the GIN cell.

Original: We evaluate the performance of the GIN on a trajectory of 5k length.
Revised: We evaluate the performance of the GIN on a trajectory of 5k length.

Original: Each point in the generated trajectories is distorted with an observation noise that follows Gaussian distribution with standard deviation σ = 0 .
Revised: Each point in the generated trajectories is distorted with an observation noise that follows Gaussian distribution with standard deviation σ = 0 .

Original: The likelihood with Gaussian distribution is calculated and maximized in the training phase.
Revised: The likelihood with Gaussian distribution is calculated and maximized in the training phase.

Original: The mean square error (MSE) of the test data for various number of training samples are depicted in figure 6.
Revised: The mean square error (MSE) of the test data for various number of train- [Table] [Figure] ing samples are depicted in figure 7.

Original: Hybrid is a graphical GNN based model Garcia Satorras et al.
Revised: Hybrid is a graphical GNN based model Garcia Satorras et al.

Original:  and DSSM Rangapuram et al.
Revised: (2019) and DSSM Rangapuram et al.

Original: (2018) is a version of LGSSM using LSTM cells.
Revised: (2018) is a version of LGSSM using LSTM cells.

Original: [Table]

Original: [Figure] Due to the non-linearity of the dynamics of this system, LGSSM has to use linearization and then use the linearized dynamics to model the transition.
Revised: Due to the non-linearity of the dynamics of this system, LGSSM has to use linearization and then use the linearized dynamics to model the transition.

Original: The DSSM model performs better for lager amount of data (>10K) because it needs to learn the dynamics.
Revised: The DSSM model performs better for lager amount of data (>10K) because it needs to learn the dynamics.

Original: The results of the Hybrid GNN and the GIN are similar, while the results of the GIN are slightly improved.
Revised: The results of the Hybrid GNN and the GIN are similar, while the results of the GIN are slightly improved.

Original: Although, the core of both models is based on the GRU cell, this enhancement may come from the structure of the GIN that learns the observation and process noises separately.
Revised: Although, the core of both models is based on the GRU cell, this enhancement may come from the structure of the GIN that learns the observation and process noises separately.

Original: Comparison with LGSSM indicates that applying non-linearity(GRU cells) to the GIN, benefits it in non-linear dynamics.

Original: The required numbers of training samples to achieve 0.1 and 0.05 MSE for the GIN are approximately 350 and 3800 samples, respectively.

Original: Inferred trajectories are in figure 1.
Revised: Inferred trajectories are in figure 1.

Original: 6.2. R EAL W ORLD D YNAMICS : M ICHIGAN NCLT DATASET
Revised: 6.2. R EAL W ORLD D YNAMICS : M ICHIGAN NCLT DATASET

Original: To evaluate the performance of the GIN on a real world dataset, the Michigan NCLT dataset CarlevarisBianco et al.
Revised: To evaluate the performance of the GIN on a real world dataset, the Michigan NCLT dataset CarlevarisBianco et al.

Original: (2016) is utilized that encompasses a collection of navigation data gathered by a segwey robot moving inside of the University of Michigan’s North Campus.
Revised: (2016) is utilized that encompasses a collection of navigation data gathered by a segwey robot moving inside of the University of Michigan’s North Campus.

Original: The states in each time, x t ∈ R 4 , comprise the position and the velocity in each direction and the observations, y t ∈ R 2 , include noisy positions.
Revised: The states in each time, x t ∈ R 4 , comprise the position and the velocity in each direction and the observations, y t ∈ R 2 , include noisy positions.

Original: The ultimate purpose is to localize the real position of the segway robot, while only the noisy
Revised: The ultimate purpose is to localize the real position of the segway robot, while only the noisy

Original: GPS observations are available.
Revised: GPS observations are available.

Original: We apply the GIN to find the current location of the segway robot.
Revised: We apply the GIN to find the current location of the segway robot.

Original: [Table]
Revised: [Table]

Original: Vanilla

Original: [Table]

Original: [Table]

Original: LGSSM

Original: [Table]

Original: Observation

Original: [Table]

Original: [Table]

Original: In this experiment, we randomly select the session 201201-22 captured in a cloudy situation with the length of 6.
Revised: In this experiment, we randomly select the session 201201-22 captured in a cloudy situation with the length of 6.

Original: Km.
Revised: Km.

Original: By sampling with 1Hz and removing the unstable
Revised: By sampling with 1Hz and removing the unstable

Original: GPS observations, 4280 time steps are achieved.
Revised: GPS observations, 4280 time steps are achieved.

Original: For the dynamics of the system, we consider a uniform motion pattern with a constant velocity (see A.7).
Revised: For the dynamics of the system, we consider a uniform motion pattern with a constant velocity (see A.7).

Original: The training procedure is completed by maximizing the likelihood with
Revised: The training procedure is completed by maximizing the likelihood with

Original: Gaussian distribution assumption.
Revised: Gaussian distribution assumption.

Original: The mean squared error of each approach for the test set are mentioned in the table 6, where the GIN ( 73 .
Revised: The mean squared error of each approach for the test set are mentioned in the table 6, where the GIN ( 73 .

Original: 12 ± 2 .
Revised: 12 ± 2 .

Original: 21 MSE) outperforms other approaches.
Revised: 21 MSE) outperforms other approaches.

Original: In summary, this experiment indicates that the
Revised: In summary, this experiment indicates that the

Original: GIN can generalize with good performance to a real world dataset.
Revised: GIN can generalize with good performance to a real world dataset.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The GIN, an approach for representation learning in both high and low dimensional SSMs, is introduced in this paper.
Revised: The GIN, an approach for representation learning in both high and low dimensional SSMs, is introduced in this paper.

Original: The data flow is conducted by Bayesian filtering-smoothing, while, due to the usage of GRU based KG and SG network, the computational issues are tackled resulting in an efficient model with numerical stable results.
Revised: The data flow is conducted by Bayesian filtering-smoothing, while, due to the usage of GRU based KG and SG network, the computational issues are tackled resulting in an efficient model with numerical stable results.

Original: In the presence of the dynamics, the GIN directly use them, otherwise it directly learns them in an end to end manner, which makes the GIN as a HW model with strong system identification abilities.
Revised: In the presence of the dynamics, the GIN directly use them, otherwise it directly learns them in an end to end manner, which makes the GIN as a HW model with strong system identification abilities.

Original: Insightful representation for the uncertainty of the predictions is incorporated in this approach, while it outperforms its counterparts including LSTMs,
Revised: Insightful representation for the uncertainty of the predictions is incorporated in this approach, while it outperforms its counterparts including LSTMs,

Original: GRUs and several generative models with variational inferences.
Revised: GRUs and several generative models with variational inferences.

Original: A A PPENDIX A. G AUSSIAN STATE SPACE MODELS
Revised: A A PPENDIX A. G AUSSIAN STATE SPACE MODELS

Original: In order to model the vectors of time series w = w 1: T =
Revised: In order to model the vectors of time series w = w 1: T =

Original: [ w 1 , ..., w T ] , Gaussian state space models (GSSMs) are commonly applied due to their filtering-smoothing ability.
Revised: [ w 1 , ..., w T ] , Gaussian state space models (GSSMs) are commonly applied due to their filtering-smoothing ability.

Original: In fact, GSSMs model the first-order Markov process on the state space x
Revised: In fact, GSSMs model the first-order Markov process on the state space x

Original: = [ x 1 , ..., x T ] , which can also include the external control input u
Revised: = [ x 1 , ..., x T ] , which can also include the external control input u

Original: = [ u 1 , ..., u T ] by multivariate normality assumption of the statep
Revised: = [ u 1 , ..., u T ] by multivariate normality assumption of the state

Original: [Equation]
Revised: [Equation]

Original: For the cases, which are not controlled via external input, B t matrix is simply 0 matrix.
Revised: For the cases, which are not controlled via external input, B t matrix is simply 0 matrix.

Original: By Defining γ t as parameters which explain how the state state changes during the time, it contains the information of F t , B t and H t which are the state transition, control and emission matrices.
Revised: By Defining γ t as parameters which explain how the state state changes during the time, it contains the information of F t , B t and H t which are the state transition, control and emission matrices.

Original: In each step, the procedure is distorted via Q and R that are process noise and observation noise, respectively.
Revised: In each step, the procedure is distorted via Q and R that are process noise and observation noise, respectively.

Original: It is common to initial the first state x 1 ∼ N ( 0 , Σ 0 ) , then the joint probability distribution of the GSSM isp
Revised: It is common to initial the first state x 1 ∼ N ( 0 , Σ 0 ) , then the joint probability distribution of the GSSM is

Original: [Equation]
Revised: [Equation]

Original: GSSMs have substantial properties that we can utilize.
Revised: GSSMs have substantial properties that we can utilize.

Original: Filtering and smoothing are among these properties which allow us to obtain the filtered and smoothed posterior based on the priors and observations.
Revised: Filtering and smoothing are among these properties which allow us to obtain the filtered and smoothed posterior based on the priors and observations.

Original: By applying classic Bayesian properties, we can have a strong tool to handle the missing data in the image imputation task.
Revised: By applying classic Bayesian properties, we can have a strong tool to handle the missing data in the image imputation task.

Original: A. F ILTERING AND S MOOTHING P ARAMETERIZATION
Revised: A. F ILTERING AND S MOOTHING P ARAMETERIZATION

Original: The idea of Kalman filter applies two iterative steps, in the former one a prediction is made by the prior state information, while in the latter one an update is done based on the obtained observation.
Revised: The idea of Kalman filter applies two iterative steps, in the former one a prediction is made by the prior state information, while in the latter one an update is done based on the obtained observation.

Original: By normality assumption of known additive process and observation noise, the filter can go through the two mentioned steps.
Revised: By normality assumption of known additive process and observation noise, the filter can go through the two mentioned steps.

Original: In the prediction step, the filter uses the transition matrix F to estimate the next priors ( x − t +1 , Σ − t +1 ) which are the estimate of the the next states without any observation.
Revised: In the prediction step, the filter uses the transition matrix F to estimate the next priors ( x − t +1 , Σ − t +1 ) which are the estimate of the the next states without any observation.

Original: [Equation]
Revised: [Equation]

Original: In the presence of new observation, the Kalman filter idea goes through the second step and modifies the predicted prior based on the new observation and emission matrix H that results in the next posterior ( x + t +1 , Σ + t +1 ) .
Revised: In the presence of new observation, the Kalman filter idea goes through the second step and modifies the predicted prior based on the new observation and emission matrix H that results in the next posterior ( x + t +1 , Σ + t +1 ) .

Original: [Equation]
Revised: [Equation]

Original: The whole observation update procedure can be considered as a weighted mean between the the next prior, that comes from state update, and new observation, where this weighting is a function of Q and
Revised: The whole observation update procedure can be considered as a weighted mean between the the next prior, that comes from state update, and new observation, where this weighting is a function of Q and

Original: R that has uncertainty nature.
Revised: R that has uncertainty nature.

Original: We derive smoothing parameterization, where the key idea is to use Markov property, which states that x t is independent of future observations w t
Revised: We derive smoothing parameterization, where the key idea is to use Markov property, which states that x t is independent of future observations w t

Original: +1: T as long as x t +1 is known.
Revised: +1: T as long as x t +1 is known.

Original: However, we are not aware of x t +1 , but there is a distribution over it.
Revised: However, we are not aware of x t +1 , but there is a distribution over it.

Original: So by conditioning on x t +1 and then marginalizing out it is possible to obtain x t conditioned on w 1: T .
Revised: So by conditioning on x t +1 and then marginalizing out it is possible to obtain x t conditioned on w 1: T .

Original: [Equation]
Revised: [Equation]

Revised: By using induction and and smoothed distribution for t + 1 :

Original: [Equation]
Revised: [Equation]

Original: we calculate the filtered two-slice distribution as follows:
Revised: we calculate the filtered two-slice distribution as follows:

Original: [Equation]
Revised: [Equation]

Original: by using Gaussian conditioning we have:
Revised: by using Gaussian conditioning we have:

Original: [Equation]
Revised: [Equation]

Original: where J t = Σ + t F t +1 [ Σ − t +1 ] − 1 .
Revised: where J t = Σ + t F t +1 [ Σ − t +1 ] − 1 .

Original: We calculate the smoothed distribution for t using the rules of iterated expectation and covariance:
Revised: We calculate the smoothed distribution for t using the rules of iterated expectation and covariance:

Original: [Equation]
Revised: [Equation]

Original: A. P ROCESS N OISE M ATRIX
Revised: A. P ROCESS N OISE M ATRIX

Original: As stated in (17)
Revised: As stated in (18)

Original: [Equation]
Revised: [Equation]

Original: we can elaborate the process noise matrix at time t in more details
Revised: we can elaborate the process noise matrix at time t in more details

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  combining (17) into (27) results in
Revised: 28) combining (18) into (28) results in

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is a function of F t , Q t − 1 , F t − 1 and H t − 1 .
Revised: which is a function of F t , Q t − 1 , F t − 1 and H t − 1 .

Original: In the GIN, ˆ F t and ˆ H t are learned by the Dynamics
Revised: In the GIN, ˆ F t and ˆ H t are learned by the Dynamics

Original: Network with the input of x + t − 1 .
Revised: Network with the input of x + t − 1 .

Original: From (19), x + t − 1 is derived as a function of both F t − 1 and H t − 1 ,  meaning the learned ˆ F t carries the information of both H t − 1 and F t − 1 .
Revised: From (20), x + t − 1 is derived as a function of both F t − 1 and H t − 1 , [Figure] meaning the learned ˆ F t carries the information of both H t − 1 and F t − 1 .

Original: Therefore, one can rewrite the equation (28) as
Revised: Therefore, one can rewrite the equation (29) as

Original: [Equation]
Revised: [Equation]

Original: where g is a nonlinear function mapping x + t − 1 and Q t − 1 to Q t .
Revised: where g is a nonlinear function mapping x + t − 1 and Q t − 1 to Q t and the graphical model for such choice of structure is in figure 9b.

Original: It is possible to go one step further and simplify x + t − 1 more, as it has Σ − t − 1 term in (19), combining it with (17) results in
Revised: It is possible to go one step further and simplify x + t − 1 more [Equation] as it has Σ − t − 1 term in (20) [Equation]

Revised: combining it with (18) results in

Original: [Equation]
Revised: [Equation]

Original: indicating that not only F t − 1 and H t − 1 , but also Q t − 1 is included in x + t − 1 , meaning that Q t can be written solely as a function of x + t − 1 .
Revised: indicating that not only F t − 1 and H t − 1 , but also Q t − 1 is included in x + t − 1 , meaning that Q t can be written solely as a function of x + t − 1 and the graphical model for such choice is in figure 9a.

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: We call g as Q Network , where g can be modeled by a MLP (31) or a recurrent network (29), based on the mentioned explanations.
Revised: We call g as Q Network , where g can be modeled by a MLP (32) or a recurrent network (30), based on the mentioned explanations.

Original: In figure 9, it is shown how the Q Network is integrated into the whole model structure.
Revised: In figure 11, it is shown how the Q Network is integrated into the whole model structure.

Original: A. O UTPUT D ISTRIBUTION
Revised: A. O UTPUT D ISTRIBUTION

Original: In the case of grayscale images, consider each pixel, y i , is one or zero with the probability of p i or 1 − p i respectively, meaning that P ( Y = y )
Revised: In the case of grayscale images, consider each pixel, y i , is one or zero with the probability of p i or 1 − p i respectively, meaning that P ( Y = y )

Original: =
Revised: =

Original: p y (1 − p ) 1 − y .
Revised: p y (1 − p ) 1 − y .

Original: By re-writing the probability equation into the exponential families form
Revised: By re-writing the probability equation into the exponential families form

Original: [Equation]
Revised: [Equation]

Original: 32) and by choosing θ = log ( p 1 − p ) and ψ ( θ ) = log (1 − p ) , we can obtain p = 1 1+ e − θ .
Revised: and by choosing θ = log ( p 1 − p ) and ψ ( θ ) = log (1 − p ) , we can obtain p = 1 1+ e − θ .

Original: It means that by considering θ as the last layer of the decoder and applying a softmax layer, p is obtained.
Revised: It means that by considering θ as the last layer of the decoder and applying a softmax layer, p is obtained.

Original: Equivalently, one can calculate the deviance between real p and estimation of it, ˆ p , which is given by
Revised: Equivalently, one can calculate the deviance between real p and estimation of it, ˆ p , which is given by

Original: [Equation]
Revised: [Equation]

Original: and minimize the deviance with respect to ˆ p as we did in (14).
Revised: and minimize the deviance with respect to ˆ p as we did in (15).

Original: Similarly, consider x , ˆ x θ and θ as the ground truth state, estimated state and the model variables respectively, where the residual follows Gaussian distribution x = ˆ x θ + ϵ ∼ N (ˆ x θ , ˆ σ θ ) , where ˆ σ θ is the estimated variance.
Revised: Similarly, consider x , ˆ x θ and θ as the ground truth state, estimated state and the model variables respectively, where the residual follows Gaussian distribution x = ˆ x θ + ϵ ∼ N (ˆ x θ , ˆ σ θ ) , where ˆ σ θ is the estimated variance.

Original: Then, the negative log likelihood is given by (34) as we obtained it in (13).
Revised: Then, the negative log likelihood is given by (35) as we obtained it in (14).

Original: [Equation]
Revised: [Equation]

Original: A. N OISE G ENERATION P ROCESS
Revised: A. N OISE G ENERATION P ROCESS

Original: In the high dimensional observation experiments, to show the noise robustness of the system, we use time correlated noise generation scheme.
Revised: In the high dimensional observation experiments, to show the noise robustness of the system, we use time correlated noise generation scheme.

Original: It makes the noise factors correlated over time by introducing a sequence of factors f t of the same length of the data sequence.
Revised: It makes the noise factors correlated over time by introducing a sequence of factors f t of the same length of the data sequence.

Original: Let f 0 ∼ U (0 , 1) and f t +1 = min ( max (0 , f t + r t ) , 1) with r t ∼ U ( − 0 .
Revised: Let f 0 ∼ U (0 , 1) and f t +1 = min ( max (0 , f t + r t ) , 1) with r t ∼ U ( − 0 .

Original: 2 , 0 . 2) , where f 0 is the initialized factor and U is the uniform distribution.
Revised: 2 , 0 . 2) , where f 0 is the initialized factor and U is the uniform distribution.

Original: Then by defining two thresholds, t 1 ∼ U (0 , 0 . 25) and t 2 ∼ U (0 . 75 , 1) , f t < t 1 are set to 0 and f t > t 2 are set to 1 and the rest are splitted linearly within the range of [0 , 1] .
Revised: Then by defining two thresholds, t 1 ∼ U (0 , 0 . 25) and t 2 ∼ U (0 . 75 , 1) , f t < t 1 are set to 0 and f t > t 2 are set to 1 and the rest are splitted linearly within the range of [0 , 1] .

Original: The t -th obtained observation is given by o t = f
Revised: The t -th obtained observation is given by o t = f

Original: t i t + (1 − f t )
Revised: t i t + (1 − f t )

Original: i pnt , where the i t is the t -th true image
Revised: i pnt , where the i t is the t -th true image

Original: and i pnt is the t -th generated pure noise.
Revised: and i pnt is the t -th generated pure noise.

Original: A. L ORENZ A TTRACTOR D YNAMICS
Revised: A. L ORENZ A TTRACTOR D YNAMICS

Original: There are three differential equations that model a Lorenz system, x the convection rate, y the horizontal temperature variation and z the vertical temperature variation.
Revised: There are three differential equations that model a Lorenz system, x the convection rate, y the horizontal temperature variation and z the vertical temperature variation.

Original: [Equation]
Revised: [Equation]

Original: where the constant values σ , ρ and β are 10 , 28 and − 83 , respectively.
Revised: where the constant values σ , ρ and β are 10 , 28 and − 83 , respectively.

Original: To construct a trajectory we use Lorenz system equations (35) with dt = 10 − 5 , then we sample from it with the step time of ∆ t = 0 .
Revised: To construct a trajectory we use Lorenz system equations (36) with dt = 10 − 5 , then we sample from it with the step time of ∆ t = 0 .

Original: Based on the equations of the system (35), the state is s t =
Revised: Based on the equations of the system (36), the state is s t =

Original: [ x t , y t , z t ] and we can write the dynamics of the system as A t and obtain the transition matrix
Revised: [ x t , y t , z t ] and we can write the dynamics of the system as A t and obtain the transition matrix

Original: Exp [ A t ] =
Revised: Exp [ A t ] =

Original: F t .
Revised: F t .

Original: To achieve this, we use the
Revised: To achieve this, we use the

Original: Taylor expansion of Exp function with 5 degrees.
Revised: Taylor expansion of Exp function with 5 degrees.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: y −  (cid:34) xyz (cid:35) , and

Original: F t =

Original: Exp [ A t ] =

Original: I +

Original: J (cid:88)

Original: j = ( A t . ∆ t ) j j ! (36) where J is the degrees of expansion and I is the identity matrix.
Revised: where J is the degrees of expansion and I is the identity matrix.

Original: For the emission matrix we use H t
Revised: For the emission matrix we use H t

Original: = I and for process and observation noise standard deviation, we use Q t = 1100 σ 2 I
Revised: = I and for process and observation noise standard deviation, we use Q t = 1100 σ 2 I

Original: and
Revised: and

Original: R t = σ 2 I , respectively.
Revised: R t = σ 2 I , respectively.

Original: A. M OVEMENT M ODEL D ETAILS FOR T HE NCLT E XPERIMENT
Revised: A. M OVEMENT M ODEL D ETAILS FOR T HE NCLT E XPERIMENT

Original: We assume that the segway robot is moved with a constant velocity, that the equations for such dynamics are given by 
Revised: We assume that the segway robot is moved with a constant velocity, that the equations for such dynamics are given by [Equation]

Original: [Equation]
Revised: [Equation]

Original: By such assumptions for the motion’s equations the transition, process noise distribution, emission and measurement noise distribution matrices can be obtained by
Revised: By such assumptions for the motion’s equations the transition, process noise distribution, emission and measurement noise distribution matrices can be obtained by

Original: [Equation]
Revised: [Equation]

Original: ∆ t    ∆ t    ∆ t 

Original:  ,

Original: H = (cid:20) 1 0 00 0 1(cid:21) ,

Original: R = λ 2 (cid:20) 1 0 0 1 (cid:21) .

Original: (38) where ∆ t = 1 since the sampling frequency is 1Hz.
Revised: where ∆ t = 1 since the sampling frequency is 1Hz.

Original: Process and measurement variance parameters, σ and λ , are unknown that the model will learn them.
Revised: Process and measurement variance parameters, σ and λ , are unknown that the model will learn them.

Original: we split the whole sequence into training, testing and validation folds with the length of 3600 ( 18 sequences of length T = 200 ) , 280 (1 sequence of length T = 280 ) and 400 (2 sequences of length T = 200 ), respectively.
Revised: we split the whole sequence into training, testing and validation folds with the length of 3600 ( 18 sequences of length T = 200 ) , 280 (1 sequence of length T = 280 ) and 400 (2 sequences of length T = 200 ), respectively.

Original: A. N ETWORK S TRUCTURE AND P ARAMETERS
Revised: A. N ETWORK S TRUCTURE AND P ARAMETERS

Original: In all experiments, Adam optimizer Kingma & Ba (2014) has been used on NVIDIA GeForce GTX 

Revised: In all experiments, Adam optimizer Kingma & Ba (2014) has been used on NVIDIA GeForce GTX

Original: Ti.
Revised: Ti.

Original: We conduct a grid search for finding the hyperparameters to rule out the possibility of the models being trained with the suboptimal hyperparameters.
Revised: We conduct a grid search for finding the hyperparameters to rule out the possibility of the models being trained with the suboptimal hyperparameters.

Original: To find the initial learning rate, by conducting a grid search between 0 .
Revised: To find the initial learning rate, by conducting a grid search between 0 .

Original: 001 and 0 .
Revised: 001 and 0 .

Original: 2 with the increment of 0 .
Revised: 2 with the increment of 0 .

Original: 005 , we select the best one among them that corresponds to the highest log-likelihood.
Revised: 005 , we select the best one among them that corresponds to the highest log-likelihood.

Original: With an initial learning rate of 0 . 006 and an exponential decay with rate of 0 . 9 every 10 epochs, we employ back propagation through time
Revised: With an initial learning rate of 0 . 006 and an exponential decay with rate of 0 . 9 every 10 epochs, we employ back propagation through time

Original: Werbos (1990) to compute the gradients as we deploy GRU cells in the structure.
Revised: Werbos (1990) to compute the gradients as we deploy GRU cells in the structure.

Original: Layer normalization technique Ba et al.
Revised: Layer normalization technique Ba et al.

Original: (2016) is used to stabilize the dynamics in the recurrent structure and normalize the filter response.
Revised: (2016) is used to stabilize the dynamics in the recurrent structure and normalize the filter response.

Original: Elu + 1 activation function, can ensure the positiveness of the diagonal elements of the process, noise and covariance matrices.
Revised: Elu + 1 activation function, can ensure the positiveness of the diagonal elements of the process, noise and covariance matrices.

Original: In order to prevent the model being stuck in the poor local minima, e.g. focusing on the reconstruction instead of learning the dynamics obtained by filtering-smoothing, we find it useful to use two training tricks for an end-to-end learning:
Revised: In order to prevent the model being stuck in the poor local minima, e.g. focusing on the reconstruction instead of learning the dynamics obtained by filtering-smoothing, we find it useful to use two training tricks for an end-to-end learning:

Original: 1- Generating time correlated noisy sequences as consecutive observations, forces the model to learn the dynamics instead of focusing on reconstruction, e.g. figure 11 and 13.
Revised: 1- Generating time correlated noisy sequences as consecutive observations, forces the model to learn the dynamics instead of focusing on reconstruction, e.g. figure 13 and 15.

Original: 2- For the first few epochs, only learn auto-encoder(MLPs) and globally learned parameters, e.g. F ( k ) and H ( k ) , but not Dynamics Network parameters α t ( x t − 1 ) .
Revised: 2- For the first few epochs, only learn auto-encoder(MLPs) and globally learned parameters, e.g. F ( k ) and H ( k ) , but not Dynamics Network parameters α t ( x t − 1 ) .

Original: All the parameters are jointly learned, afterwards.
Revised: All the parameters are jointly learned, afterwards.

Original: This allows the system to learn good embedding and meaningful latent vectors at first, then learns how to employ K different dynamics variables.
Revised: This allows the system to learn good embedding and meaningful latent vectors at first, then learns how to employ K different dynamics variables.

Original: In the lack of dynamics, for the low dimensional observations we use K = 5 , while for the high dimensional observations we use K = 15 as they need to learn more complex dynamics.
Revised: In the lack of dynamics, for the low dimensional observations we use K = 5 , while for the high dimensional observations we use K = 15 as they need to learn more complex dynamics.

Original: In general, if the GIN is flexible enough, tuning the parameters is not difficult as the GIN is capable to learn how to prune unused elements by the Dynamics Network .
Revised: In general, if the GIN is flexible enough, tuning the parameters is not difficult as the GIN is capable to learn how to prune unused elements by the Dynamics Network .

Original: To prevent the model being stuck into mode collapse, we provided two solutions:
Revised: To prevent the model being stuck into mode collapse, we provided two solutions:

Original: 1- By introducing k sets of F k , H k , where each set of F k , H k models different dynamics, we introduce a loss term with a small constant factor which tries to increase the distance of each pair of F k , H k set.
Revised: 1- By introducing k sets of F k , H k , where each set of F k , H k models different dynamics, we introduce a loss term with a small constant factor which tries to increase the distance of each pair of F k , H k set.

Original: Intuitively, the presence of different dynamics can easily modify the states in each update.
Revised: Intuitively, the presence of different dynamics can easily modify the states in each update.

Original: We found this method as a potential solution to prevent the model go through the mode collapse.
Revised: We found this method as a potential solution to prevent the model go through the mode collapse.

Original: 2- Considering the negative distance of consecutive pairs of states as additional loss term with a small constant factor (the distance can be considered as euclidean difference of mean or KL of two consecutive states).
Revised: 2- Considering the negative distance of consecutive pairs of states as additional loss term with a small constant factor (the distance can be considered as euclidean difference of mean or KL of two consecutive states).

Original: Intuitively, this solution is forcing the states to not have overlap with each other and impose them to change in each update step.
Revised: Intuitively, this solution is forcing the states to not have overlap with each other and impose them to change in each update step.

Original: In the simulation results, we have used the first option.
Revised: In the simulation results, we have used the first option.

Original: A.8. P ROPOSED ARCHITECTURE FOR HIGH AND LOW DIMENSIONAL OBSERVATIONS
Revised: A.8. P ROPOSED ARCHITECTURE FOR HIGH AND LOW DIMENSIONAL OBSERVATIONS

Original: The proposed structure to deal with high dimensional observations in the lack of the dynamics(the first three experiments in the paper) is shown in figure 8.
Revised: The proposed structure to deal with high dimensional observations in the lack of the dynamics(the first three experiments in the paper) is shown in figure 10.

Original: While, the proposed structure to handle low dimensional observations in the presence of the dynamics(the last two experiments in the paper) is shown in the figure 9.
Revised: While, the proposed structure to handle low dimensional observations in the presence of the dynamics(the last two experiments in the paper) is shown in the figure 11.

Original: A.8. E MPIRICAL RUNNING TIMES AND PARAMETERS
Revised: A.8. E MPIRICAL RUNNING TIMES AND PARAMETERS

Original: We present the number of parameters of the utilized cell structures in our experiments and their corresponding empirical running times for 1 epoch in the table 7 and 8.
Revised: We present the number of parameters of the utilized cell structures in our experiments and their corresponding empirical running times for 1 epoch in the table 7 and 8.

Original: In the first row of each model structure in the high dimensional experiments, we set the number of their parameters approximately equal to our GIN to indicate the outperformance of the GIN with the same number of the parameters, i.e. tables 1, 2 and 5, while we include the empirical running time and parameters for more complex structures as well.
Revised: In the first row of each model structure in the high dimensional experiments, we set the number of their parameters approximately equal to our GIN to indicate the outperformance of the GIN with the same number of the parameters, i.e. tables 1, 2 and 5, while we include the empirical running time and parameters for more complex structures as well.

Original: Extra running time of EM-variational approaches, like KVAE, is due to employing classic Bayesian equations because it increases the running time substantially when dealing with higher dimensional observations, however, the GIN circumvent this difficulty.
Revised: Extra running time of EM-variational approaches, like KVAE, is due to employing classic Bayesian equations because it increases the running time substantially when dealing with higher dimensional observations, however, the GIN circumvent this difficulty.

Original: The number of the parameters of the GIN are noticeably lower than other memory cells, e.g. LSTM and GRU, and
Revised: The number of the parameters of the GIN are noticeably lower than other memory cells, e.g. LSTM and GRU, and

Original: EM-variational methods as we convert high dimensional sparse covariance matrices into lower dimensional covariance matrices by employing convolutional operator.
Revised: EM-variational methods as we convert high dimensional sparse covariance matrices into lower dimensional covariance matrices by employing convolutional operator.

Original: It allows us to reduce the parameters of internal GRU cells quadratically.
Revised: It allows us to reduce the parameters of internal GRU cells quadratically.

Original: A.8. Q UALITATIVE COMPARISON OF THE GIN TO RECENT RELATED WORK .
Revised: A.8. Q UALITATIVE COMPARISON OF THE GIN TO RECENT RELATED WORK .

Original: In table 9, we make a comparison to show whether algorithms are able to handle high and low dimensional observations, learn dynamics, use available-partial dynamics, estimate state appropriately, provide model’s uncertainty estimates handling noisy data, handle missing data and perform direct optimization.
Revised: In table 9, we make a comparison to show whether algorithms are able to handle high and low dimensional observations, learn dynamics, use available-partial dynamics, estimate state appropriately, provide model’s uncertainty estimates handling noisy data, handle missing data and perform direct optimization.

Original: Classic LGSSMs, e.g. EKF and UKF, work based on the linearization of the transition and emission equations and apply classic Bayesian updates over the linearized system with respect to the states.
Revised: Classic LGSSMs, e.g. EKF and UKF, work based on the linearization of the transition and emission equations and apply classic Bayesian updates over the linearized system with respect to the states.

Original: In other words, ( F , H ) in the classic LGSSMs are not data-deriven nor trainable.
Revised: In other words, ( F , H ) in the classic LGSSMs are not data-deriven nor trainable.

Original: Despite classic LGSSMs, in the GIN we use a data-driven based network to learn dynamics, i.e.
Revised: Despite classic LGSSMs, in the GIN we use a data-driven based network to learn dynamics, i.e.

Original: Dynamics Network in the paper.
Revised: Dynamics Network in the paper.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: ✓ ✓ SSIRuhe&Forré(2021) × ✓ × ✓ ✓ ✓ ✓ ✓

Original: LGSSM × ✓ × / ✓ ✓ ✓ ✓ ✓ ✓ GIN ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Original: A.8. S INGLE -P ENDULUM AND D OUBLE -P ENDULUM E XPERIMENTS 
Revised: A.8. S INGLE -P ENDULUM AND D OUBLE -P ENDULUM E XPERIMENTS D ATA G ENERATION

Original: D ATA G ENERATION .

Original: Dataset consists of 1000 train, 100 valid and 100 test sequences with the length of 150.
Revised: Dataset consists of 1000 train, 100 valid and 100 test sequences with the length of 150.

Original: The sequences are distorted via generated noise, while in the informed imputation task half of the images are removed and boolean flags indicating the availability of the observations are passed to the cell instead.
Revised: The sequences are distorted via generated noise, while in the informed imputation task half of the images are removed and boolean flags indicating the availability of the observations are passed to the cell instead.

Original: If the imputation task is in uninformed type, black images are considered as the observations instead of informing the cell with boolean flags.
Revised: If the imputation task is in uninformed type, black images are considered as the observations instead of informing the cell with boolean flags.

Original: E NCODER /D ECODER AND THE D YNAMICS N ETWORK A RCHITECTURE
Revised: E NCODER /D ECODER AND THE D YNAMICS N ETWORK A RCHITECTURE

Original: To design the dynamics network, we use a MLP including 60 hidden units with Relu activation function and a softmax activation for the last layer.
Revised: To design the dynamics network, we use a MLP including 60 hidden units with Relu activation function and a softmax activation for the last layer.

Original: The state mean, with size of n , and number of the bases, with size of k , are the input and output of the dynamics, respectively.
Revised: The state mean, with size of n , and number of the bases, with size of k , are the input and output of the dynamics, respectively.

Original: The structures of the encoder and decoder are in the table 10.
Revised: The structures of the encoder and decoder are in the table 10.

Original: In the table 10, m is the latent observation dimension that various values for this parameter are taken into account in the results.
Revised: In the table 10, m is the transferred observation dimension that various values for this parameter are taken into account in the results.

Original: In the state estimation tasks, out dim is 4 and 8 for the single-pendulum and double-pendulum experiment, respectively.
Revised: In the state estimation tasks, out dim is 4 and 8 for the single-pendulum and double-pendulum experiment, respectively.

Original: For the imputation task, number of the hidden units of the KG and SG network is set to 40 and 30, respectively.
Revised: For the imputation task, number of the hidden units of the KG and SG network is set to 40 and 30, respectively.

Original: The convolutional layer applied over the covariance matrix has 8 filters with kernel size of 5.
Revised: The convolutional layer applied over the covariance matrix has 8 filters with kernel size of 5.

Original: [Table]
Revised: A.8. L ORENZ A TTRACTOR AND NCLT E XPERIMENTS

Original: if
Revised: In these two experiments that we have the knowledge of the dynamics

Original: [Table]
Revised: [Table]

Original: () fully connected:fully connected:6 × 6 Trns Conv, 16, stride 4 ×w : fully connected: m , linear activation

Original: LayerNormalizer() r : fully connected, Elu + 1 activation 4 × 4 Trns Conv, 12, stride 2 ×

Original: LayerNormalizer() o + i : 1 × 1 Trns Conv, stride 1 × 1 , softmax A.8. L ORENZ A TTRACTOR AND NCLT E XPERIMENTS In these two experiments that we have the knowledge of the dynamics, we employ a fully connected with the observations as its input and output dimension of 3 and 2 for Lorenz attractor and NCLT experiments, respectively, to obtain the observation noise, r .
Revised: we employ a fully connected with the observations as its input and output dimension of 3 and 2 for Lorenz attractor and NCLT experiments, respectively, to obtain the observation noise, r .

Original: The activation function is Elu + 1.
Revised: The activation function is Elu + 1.

Original: Similarly another fully connected with the posterior state as its input and output dimension of 3 andfor Lorenz attractor and NCLT experiments, respectively, to attain the uncertainty estimates, o + σ .
Revised: Similarly another fully connected with the posterior state as its input and output dimension of 3 andfor Lorenz attractor and NCLT experiments, respectively, to attain the uncertainty estimates, o + σ .

Original: To estimate the process noise matrix, a fully connect with the posterior state as the input and Elu +activation function is used.
Revised: To estimate the process noise matrix, a fully connect with the posterior state as the input and Elu +activation function is used.

Original: Similarly, a GRU cell that maps the posterior states to the process noise matrix with 10 hidden units can be used.
Revised: Similarly, a GRU cell that maps the posterior states to the process noise matrix with 10 hidden units can be used [Table] [Table] 25k

Original: A. V ISUALIZATION AND T HE I MPUTATION
Revised: [Table] [Table] [Table] [Table]

Revised: ✓ HybridGNNGarciaSatorrasetal.(2019) × ✓ × ✓ ✓ × ✓ ✓ KalmanNetRevachetal.(2021)

Revised: × ✓ × ✓ ✓ × ✓ ✓ SSIRuhe&Forré(2021) × ✓ × ✓ ✓ ✓ ✓ ✓

Revised: LGSSM × ✓ × / ✓ ✓ ✓ ✓ ✓ ✓ GIN ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Revised: A. V ISUALIZATION AND T HE I MPUTATION

Original: Graphical results of informed, uninformed and noisy observations for image imputation task for both single and double pendulum experiments can be found in 10, 11, 12 and 13 figures.
Revised: Graphical results of informed, uninformed and noisy observations for image imputation task for both single and double pendulum experiments can be found in 12, 13, 14 and 15 figures.

Original: Inference for the trained smoothing and filtering distributions of all high dimensional experiments are in14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36 and 37 figures, where we generated samples from the smoothing distribution, f ( x t | w 1: T ) , and the filtering distribution, f ( x t | w 1: t ) .
Revised: Inference for the trained smoothing and filtering distributions of all high dimensional experiments are in 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38 and 39 figures, where we generated samples from the smoothing distribution, f ( x t | w 1: T ) , and the filtering distribution, f ( x t | w 1: t ) .

Original: Then we fit density on the generated samples.
Revised: Then we fit density on the generated samples.

Original: This visualization shows the effectiveness of the GIN in reducing the uncertainty of the estimates compare to LGSSM and KVAE.
Revised: This visualization shows the effectiveness of the GIN in reducing the uncertainty of the estimates compare to LGSSM and KVAE.

Original: Finally, the results of NCLT experiment are in figure 38.
Revised: Finally, the results of NCLT experiment are in figure 47. [Table] [Table]

Original: [Figure]
Revised: 6 Conv, 12, max pooling 2 × 2 , stride 2 ×o + x : fully connected: out dim , linear activation LayerNormalizer() o +Σ : fully connected, Elu + 1 activation 4 × 4 Conv, 12, max pooling 2 × 2 , stride 2 ×if imputation task : LayerNormalizer() fully connected:fully connected:6 × 6 Trns Conv, 16, stride 4 ×w : fully connected: m , linear activation LayerNormalizer() r : fully connected, Elu + 1 activation 4 × 4 Trns Conv, 12, stride 2 ×

Revised: LayerNormalizer() o

Original:  [Figure]
Revised: [Equation] [Equation] [Figure] [Figure]

Original: We calculate the sample mean and fit a distribution on the samples for further visualization and comparison purpose.
Revised: We calculate the sample mean and fit a distribution on the samples for further visualization and comparison purpose.

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. MSE R ESULTS FOR T HE S TATE E STIMATION T ASK
Revised: A. MSE R ESULTS FOR T HE S TATE E STIMATION AND E STIMATED KG-SG

Revised: We compare the learned KG-SG matrices via the GRU cells with their corresponding ground truth for the first 100 time steps of the low dimensional experiments.

Revised: We calculate the element-wise squared difference of the learned KG and its ground truth, ∆KG t =

Revised: Tr (cid:0) ( ˆKG t − KG t ) T ( ˆKG t − KG t ) (cid:1) , and take the average of all ∆KG t , while similar procedure holds for the SG.

Revised: The results are provided in table 11.

Revised: NCLT dynamics refer to the situation, where are given the dynamics form

Original:  The MSE results for the single and double pendulum experiments are in the table 11 and 12.
Revised: [Table] The MSE results for the single and double pendulum experiments are in the table 12 and 13.

Original: In addition to (6), where F matrix includes the effects of the process noise, two other mentioned solutions introduced in section 4, are included in the MSE results as well.
Revised: In addition to (7), where F matrix includes the effects of the process noise, two other mentioned solutions introduced in section 4, are included in the MSE results as well.

Original: Using GRU cell and MLP for mapping x + , as their input,to Q , as their output, where the former one is shown by GRU( Q ) and the latter one by MLP( Q ) in the tables.
Revised: Using GRU cell and MLP for mapping x + , as their input,

Original: [Table]
Revised: [Table]

Original: LGSSM filter( m = 20 , n = 40 , K = 10 ) 0.084 ± 0.
Revised: LGSSM filter( m = 20 , n = 40 , K = 10 ) 0.084 ± 0.

Original: 0.082 ± 0. 0.082 ± 0.
Revised: 0.082 ± 0. 0.082 ± 0.

Original: LGSSM filter( m = 20 , n = 40 , K = 15 ) 0.083 ± 0.
Revised: LGSSM filter( m = 20 , n = 40 , K = 15 ) 0.083 ± 0.

Original: 0.081 ± 0.
Revised: 0.081 ± 0.

Original: 0.080 ± 0.
Revised: 0.080 ± 0.

Original: LGSSM filter( m = 20 , n = 60 , K = 10 ) 0.079 ± 0.
Revised: LGSSM filter( m = 20 , n = 60 , K = 10 ) 0.079 ± 0.

Original: 0.078 ± 0.
Revised: 0.078 ± 0.

Original: 0.076 ± 0.
Revised: 0.076 ± 0.

Original: LGSSM filter( m = 20 , n = 60 , K = 15 ) 0.077 ± 0.
Revised: LGSSM filter( m = 20 , n = 60 , K = 15 ) 0.077 ± 0.

Original: 0.075 ± 0.
Revised: 0.075 ± 0.

Original: 0.074 ± 0.
Revised: 0.074 ± 0.

Original: LGSSM smooth( m = 15 , n = 30 , K = 10 ) 0.086 ± 0.
Revised: LGSSM smooth( m = 15 , n = 30 , K = 10 ) 0.086 ± 0.

Original: 0.083 ± 0.
Revised: 0.083 ± 0.

Original: 0.084 ± 0.
Revised: 0.084 ± 0.

Original: LGSSM smooth( m = 15 , n = 30 , K = 15 ) 0.085 ± 0.
Revised: LGSSM smooth( m = 15 , n = 30 , K = 15 ) 0.085 ± 0.

Original: 0.084 ± 0.
Revised: 0.084 ± 0.

Original: 0.083 ± 0.
Revised: 0.083 ± 0.

Original: LGSSM smooth( m = 15 , n = 45 , K = 10 ) 0.081 ± 0.
Revised: LGSSM smooth( m = 15 , n = 45 , K = 10 ) 0.081 ± 0.

Original: 0.080 ± 0.
Revised: 0.080 ± 0.

Original: 0.079 ± 0.
Revised: 0.079 ± 0.

Original: LGSSM smooth( m = 15 , n = 45 , K = 15 ) 0.081 ± 0.
Revised: LGSSM smooth( m = 15 , n = 45 , K = 15 ) 0.081 ± 0.

Original: 0.078 ± 0.
Revised: 0.078 ± 0.

Original: 0.077 ± 0.
Revised: 0.077 ± 0.

Original: LGSSM smooth( m = 20 , n = 40 , K = 10 ) 0.082 ± 0.
Revised: LGSSM smooth( m = 20 , n = 40 , K = 10 ) 0.082 ± 0.

Original: 0.078 ± 0.
Revised: 0.078 ± 0.

Original: 0.076 ± 0.
Revised: 0.076 ± 0.

Original: LGSSM smooth( m = 20 , n = 40 , K = 15 ) 0.080 ± 0.
Revised: LGSSM smooth( m = 20 , n = 40 , K = 15 ) 0.080 ± 0.

Original: 0.076 ± 0.
Revised: 0.076 ± 0.

Original: 0.074 ± 0.
Revised: 0.074 ± 0.

Original: LGSSM smooth( m = 20 , n = 60 , K = 10 ) 0.076 ± 0.
Revised: LGSSM smooth( m = 20 , n = 60 , K = 10 ) 0.076 ± 0.

Original: 0.073 ± 0.
Revised: 0.073 ± 0.

Original: 0.070 ± 0.
Revised: 0.070 ± 0.

Original: LGSSM smooth( m = 20 , n = 60 , K = 15 ) 0.073 ± 0.
Revised: LGSSM smooth( m = 20 , n = 60 , K = 15 ) 0.073 ± 0.

Original: 0.071 ± 0.
Revised: 0.071 ± 0.

Original: 0.068 ± 0.
Revised: 0.068 ± 0.

Original: GIN filter( m = 15 , n = 30 , K = 10 ) 0.078 ± 0.
Revised: GIN filter( m = 15 , n = 30 , K = 10 ) 0.078 ± 0.

Original: 0.076 ± 0.
Revised: 0.076 ± 0.

Original: 0.075 ± 0.
Revised: 0.075 ± 0.

Original: GIN filter( m = 15 , n = 30 , K = 15 ) 0.078 ± 0.
Revised: GIN filter( m = 15 , n = 30 , K = 15 ) 0.078 ± 0.

Original: 0.075 ± 0.
Revised: 0.075 ± 0.

Original: 0.074 ± 0.
Revised: 0.074 ± 0.

Original: GIN filter( m = 15 , n = 45 , K = 10 ) 0.074 ± 0.
Revised: GIN filter( m = 15 , n = 45 , K = 10 ) 0.074 ± 0.

Original: 0.073 ± 0.
Revised: 0.073 ± 0.

Original: 0.072 ± 0.
Revised: 0.072 ± 0.

Original: GIN filter( m = 15 , n = 45 , K = 15 ) 0.073 ± 0.
Revised: GIN filter( m = 15 , n = 45 , K = 15 ) 0.073 ± 0.

Original: 0.074 ± 0.
Revised: 0.074 ± 0.

Original: 0.071 ± 0.
Revised: 0.071 ± 0.

Original: GIN filter( m = 20 , n = 40 , K = 10 ) 0.072 ± 0.
Revised: GIN filter( m = 20 , n = 40 , K = 10 ) 0.072 ± 0.

Original: 0.072 ± 0.
Revised: 0.072 ± 0.

Original: 0.070 ± 0.
Revised: 0.070 ± 0.

Original: GIN filter( m = 20 , n = 40 , K = 15 ) 0.071 ± 0.
Revised: GIN filter( m = 20 , n = 40 , K = 15 ) 0.071 ± 0.

Original: 0.071 ± 0.
Revised: 0.071 ± 0.

Original: 0.071 ± 0.
Revised: 0.071 ± 0.

Original: GIN filter( m = 20 , n = 60 , K = 10 ) 0.067 ± 0.
Revised: GIN filter( m = 20 , n = 60 , K = 10 ) 0.067 ± 0.

Original: 0.066 ± 0.
Revised: 0.066 ± 0.

Original: 0.065 ± 0.
Revised: 0.065 ± 0.

Original: GIN filter( m = 20 , n = 60 , K = 15 ) 0.065 ± 0.
Revised: GIN filter( m = 20 , n = 60 , K = 15 ) 0.065 ± 0.

Original: 0.064 ± 0.
Revised: 0.064 ± 0.

Original: 0.063 ± 0.
Revised: 0.063 ± 0.

Original: GIN smooth( m = 15 , n = 30 , K = 10 ) 0.071 ± 0.
Revised: GIN smooth( m = 15 , n = 30 , K = 10 ) 0.071 ± 0.

Original: 0.070 ± 0.
Revised: 0.070 ± 0.

Original: 0.068 ± 0.
Revised: 0.068 ± 0.

Original: GIN smooth( m = 15 , n = 30 , K = 15 ) 0.070 ± 0.
Revised: GIN smooth( m = 15 , n = 30 , K = 15 ) 0.070 ± 0.

Original: 0.068 ± 0.
Revised: 0.068 ± 0.

Original: 0.068 ± 0.
Revised: 0.068 ± 0.

Original: GIN smooth( m = 15 , n = 45 , K = 10 ) 0.065 ± 0.
Revised: GIN smooth( m = 15 , n = 45 , K = 10 ) 0.065 ± 0.

Original: 0.065 ± 0.
Revised: 0.065 ± 0.

Original: 0.064 ± 0.
Revised: 0.064 ± 0.

Original: GIN smooth( m = 15 , n = 45 , K = 15 ) 0.064 ± 0.
Revised: GIN smooth( m = 15 , n = 45 , K = 15 ) 0.064 ± 0.

Original: 0.066 ± 0.
Revised: 0.066 ± 0.

Original: 0.063 ± 0.
Revised: 0.063 ± 0.

Original: GIN smooth( m = 20 , n = 40 , K = 10 ) 0.064 ± 0.
Revised: GIN smooth( m = 20 , n = 40 , K = 10 ) 0.064 ± 0.

Original: 0.065 ± 0.
Revised: 0.065 ± 0.

Original: 0.062 ± 0.
Revised: 0.062 ± 0.

Original: GIN smooth( m = 20 , n = 40 , K = 15 ) 0.063 ± 0.
Revised: GIN smooth( m = 20 , n = 40 , K = 15 ) 0.063 ± 0.

Original: 0.064 ± 0.
Revised: 0.064 ± 0.

Original: 0.063 ± 0.
Revised: 0.063 ± 0.

Original: GIN smooth( m = 20 , n = 60 , K = 10 ) 0.059 ± 0.
Revised: GIN smooth( m = 20 , n = 60 , K = 10 ) 0.059 ± 0.

Original: 0.061 ± 0.
Revised: 0.061 ± 0.

Original: 0.057 ± 0.
Revised: 0.057 ± 0.

Original: GIN smooth( m = 20 , n = 60 , K = 15 ) 0.058 ± 0.
Revised: GIN smooth( m = 20 , n = 60 , K = 15 ) 0.058 ± 0.

Original: 0.057 ± 0.
Revised: 0.057 ± 0.

Original: 0.056 ± 0.
Revised: 0.056 ± 0.

Original: [Table]
Revised: [Table]

Original: 0.137 ± 0.
Revised: 0.137 ± 0.

Original: 0.138 ± 0.
Revised: 0.138 ± 0.

Original: LGSSM filter( m = 20 , n = 40 , K = 15 ) 0.141 ± 0.
Revised: LGSSM filter( m = 20 , n = 40 , K = 15 ) 0.141 ± 0.

Original: 0.137 ± 0.
Revised: 0.137 ± 0.

Original: 0.136 ± 0.
Revised: 0.136 ± 0.

Original: LGSSM filter( m = 20 , n = 60 , K = 10 ) 0.129 ± 0.
Revised: LGSSM filter( m = 20 , n = 60 , K = 10 ) 0.129 ± 0.

Original: 0.126 ± 0.
Revised: 0.126 ± 0.

Original: 0.122 ± 0.
Revised: 0.122 ± 0.

Original: LGSSM filter( m = 20 , n = 60 , K = 15 ) 0.127 ± 0.
Revised: LGSSM filter( m = 20 , n = 60 , K = 15 ) 0.127 ± 0.

Original: 0.124 ± 0.
Revised: 0.124 ± 0.

Original: 0.119 ± 0.
Revised: 0.119 ± 0.

Original: LGSSM smooth( m = 15 , n = 30 , K = 10 ) 0.147 ± 0.
Revised: LGSSM smooth( m = 15 , n = 30 , K = 10 ) 0.147 ± 0.

Original: 0.148 ± 0.
Revised: 0.148 ± 0.

Original: 0.144 ± 0.
Revised: 0.144 ± 0.

Original: LGSSM smooth( m = 15 , n = 30 , K = 15 ) 0.146 ± 0.
Revised: LGSSM smooth( m = 15 , n = 30 , K = 15 ) 0.146 ± 0.

Original: 0.146 ± 0.
Revised: 0.146 ± 0.

Original: 0.142 ± 0.
Revised: 0.142 ± 0.

Original: LGSSM smooth( m = 15 , n = 45 , K = 10 ) 0.139 ± 0.
Revised: LGSSM smooth( m = 15 , n = 45 , K = 10 ) 0.139 ± 0.

Original: 0.136 ± 0.
Revised: 0.136 ± 0.

Original: 0.133 ± 0.
Revised: 0.133 ± 0.

Original: LGSSM smooth( m = 15 , n = 45 , K = 15 ) 0.137 ± 0.
Revised: LGSSM smooth( m = 15 , n = 45 , K = 15 ) 0.137 ± 0.

Original: 0.135 ± 0.
Revised: 0.135 ± 0.

Original: 0.133 ± 0.
Revised: 0.133 ± 0.

Original: LGSSM smooth( m = 20 , n = 40 , K = 10 ) 0.136 ± 0.
Revised: LGSSM smooth( m = 20 , n = 40 , K = 10 ) 0.136 ± 0.

Original: 0.131 ± 0.
Revised: 0.131 ± 0.

Original: 0.132 ± 0.
Revised: 0.132 ± 0.

Original: LGSSM smooth( m = 20 , n = 40 , K = 15 ) 0.134 ± 0.
Revised: LGSSM smooth( m = 20 , n = 40 , K = 15 ) 0.134 ± 0.

Original: 0.129 ± 0.
Revised: 0.129 ± 0.

Original: 0.129 ± 0.
Revised: 0.129 ± 0.

Original: LGSSM smooth( m = 20 , n = 60 , K = 10 ) 0.123 ± 0.
Revised: LGSSM smooth( m = 20 , n = 60 , K = 10 ) 0.123 ± 0.

Original: 0.116 ± 0.
Revised: 0.116 ± 0.

Original: 0.115 ± 0.
Revised: 0.115 ± 0.

Original: LGSSM smooth( m = 20 , n = 60 , K = 15 ) 0.120 ± 0.
Revised: LGSSM smooth( m = 20 , n = 60 , K = 15 ) 0.120 ± 0.

Original: 0.112 ± 0.
Revised: 0.112 ± 0.

Original: 0.108 ± 0.
Revised: 0.108 ± 0.

Original: GIN filter( m = 15 , n = 30 , K = 10 ) 0.126 ± 0.
Revised: GIN filter( m = 15 , n = 30 , K = 10 ) 0.126 ± 0.

Original: 0.125 ± 0. 0.125 ± 0.
Revised: 0.125 ± 0. 0.125 ± 0.

Original: GIN filter( m = 15 , n = 30 , K = 15 ) 0.124 ± 0.
Revised: GIN filter( m = 15 , n = 30 , K = 15 ) 0.124 ± 0.

Original: 0.124 ± 0.
Revised: 0.124 ± 0.

Original: 0.121 ± 0.
Revised: 0.121 ± 0.

Original: GIN filter( m = 15 , n = 45 , K = 10 ) 0.115 ± 0.
Revised: GIN filter( m = 15 , n = 45 , K = 10 ) 0.115 ± 0.

Original: 0.114 ± 0.
Revised: 0.114 ± 0.

Original: 0.110 ± 0.
Revised: 0.110 ± 0.

Original: GIN filter( m = 15 , n = 45 , K = 15 ) 0.114 ± 0.
Revised: GIN filter( m = 15 , n = 45 , K = 15 ) 0.114 ± 0.

Original: 0.112 ± 0.
Revised: 0.112 ± 0.

Original: 0.110 ± 0.
Revised: 0.110 ± 0.

Original: GIN filter( m = 20 , n = 40 , K = 10 ) 0.113 ± 0.
Revised: GIN filter( m = 20 , n = 40 , K = 10 ) 0.113 ± 0.

Original: 0.111 ± 0. 0.111 ± 0.
Revised: 0.111 ± 0. 0.111 ± 0.

Original: GIN filter( m = 20 , n = 40 , K = 15 ) 0.111 ± 0.
Revised: GIN filter( m = 20 , n = 40 , K = 15 ) 0.111 ± 0.

Original: 0.109 ± 0.
Revised: 0.109 ± 0.

Original: 0.108 ± 0.
Revised: 0.108 ± 0.

Original: GIN filter( m = 20 , n = 60 , K = 10 ) 0.099 ± 0.
Revised: GIN filter( m = 20 , n = 60 , K = 10 ) 0.099 ± 0.

Original: 0.094 ± 0.
Revised: 0.094 ± 0.

Original: 0.095 ± 0.
Revised: 0.095 ± 0.

Original: GIN filter( m = 20 , n = 60 , K = 15 ) 0.097 ± 0.
Revised: GIN filter( m = 20 , n = 60 , K = 15 ) 0.097 ± 0.

Original: 0.093 ± 0.
Revised: 0.093 ± 0.

Original: 0.091 ± 0.
Revised: 0.091 ± 0.

Original: GIN smooth( m = 15 , n = 30 , K = 10 ) 0.115 ± 0.
Revised: GIN smooth( m = 15 , n = 30 , K = 10 ) 0.115 ± 0.

Original: 0.116 ± 0.
Revised: 0.116 ± 0.

Original: 0.113 ± 0.
Revised: 0.113 ± 0.

Original: GIN smooth( m = 15 , n = 30 , K = 15 ) 0.113 ± 0.
Revised: GIN smooth( m = 15 , n = 30 , K = 15 ) 0.113 ± 0.

Original: 0.113 ± 0.
Revised: 0.113 ± 0.

Original: 0.112 ± 0.
Revised: 0.112 ± 0.

Original: GIN smooth( m = 15 , n = 45 , K = 10 ) 0.105 ± 0.
Revised: GIN smooth( m = 15 , n = 45 , K = 10 ) 0.105 ± 0.

Original: 0.101 ± 0. 0.101 ± 0.
Revised: 0.101 ± 0. 0.101 ± 0.

Original: GIN smooth( m = 15 , n = 45 , K = 15 ) 0.102 ± 0.
Revised: GIN smooth( m = 15 , n = 45 , K = 15 ) 0.102 ± 0.

Original: 0.100 ± 0.
Revised: 0.100 ± 0.

Original: 0.098 ± 0.
Revised: 0.098 ± 0.

Original:  GIN smooth( m = 20 , n = 40 , K = 10 )0.101 ± 0.
Revised: [Table] GIN smooth( m = 20 , n = 40 , K = 10 ) 0.

Original: 0.098 ± 0.
Revised: [Table] [Table] [Table]

Original: 0.094 ± 0. GIN smooth( m = 20 , n = 40 , K = 15 ) 0.098 ± 0. 
Revised: [Table] GIN smooth( m = 20 , n = 40 , K = 15 ) 0.098 ± 0. [Table]

Original: 0.095 ± 0.
Revised: [Table]

Original: 0.092 ± 0. GIN smooth( m = 20 , n = 60 , K = 10 ) 0.086 ± 0. 
Revised: [Table] GIN smooth( m = 20 , n = 60 , K = 10 ) 0.086 ± 0. [Table]

Original: 0.081 ± 0.
Revised: 0.081 ± 0.

Original: 0.079 ± 0.
Revised: 0.079 ± 0.

Original: GIN smooth( m = 20 , n = 60 , K = 15 ) 0.083 ± 0.
Revised: GIN smooth( m = 20 , n = 60 , K = 15 ) 0.083 ± 0.

Original: 0.079 ± 0.
Revised: 0.079 ± 0.

Original: 0.076 ± 0.
Revised: 0.076 ± 0.

Original: A. A LGORITHMS
Revised: A. A LGORITHMS

Original: [Table]
Revised: [Table]

Original: Input: Ground Truth gt 1: T , Observations o 1: T , last posteriors ( x +1: T , Σ
Revised: Input: Ground Truth gt 1: T , Observations o 1: T , last posteriors ( x +1: T , Σ

Original: +1: T ) , initial posterior ( x +0 , Σ +0 )
Revised: +1: T ) , initial posterior ( x +0 , Σ +0 )

Original: α 1: T = Dynamics Network ( x
Revised: α 1: T = Dynamics Network ( x

Original: +0: T − 1 )
Revised: +0: T − 1 )

Original: Obtain ˆ F 1: T and ˆ H 1: T by (12) ( x − 1: T , Σ − 1: T ) =
Revised: Obtain ˆ F 1: T and ˆ H 1: T by (13) ( x − 1: T , Σ − 1: T ) =

Original: Prediction Step (( x +0: T − 1 , Σ +0: T − 1 ) , ˆ F 1: T ) ( w 1: T , r 1: T ) =
Revised: Prediction Step (( x +0: T − 1 , Σ +0: T − 1 ) , ˆ F 1: T ) ( w 1: T , r 1: T ) =

Original: encoder ( o 1: T )
Revised: encoder ( o 1: T )

Original: K 1: T = Σ − 1: T
Revised: K 1: T = Σ − 1: T

Original: ˆ H 1: T M 1: T M T 1: T ,
Revised: ˆ H 1: T M 1: T M T 1: T ,

Original: M 1: T = GRU KG ( Conv2D ( Σ − 1: T ) , r 1: T )
Revised: M 1: T = GRU KG ( Conv2D ( Σ − 1: T ) , r 1: T )

Original: J 1: T = Σ +1:
Revised: J 1: T = Σ +1:

Original: T ˆ F T 1: T N 1: T N T 1: T ,
Revised: T ˆ F T 1: T N 1: T N T 1: T ,

Original: N 1: T =
Revised: N 1: T =

Original: GRU SG ( Conv2D ( Σ − 1: T ) )
Revised: GRU SG ( Conv2D ( Σ − 1: T ) )

Original: ( x +1: T , Σ +1: T ) =
Revised: ( x +1: T , Σ +1: T ) =

Original: Filtering Step ( x − 1: T , Σ − 1: T , K 1: T , w 1: T , ˆ H 1: T ) ( x 1: T | T , Σ 1: T | T ) =
Revised: Filtering Step ( x − 1: T , Σ − 1: T , K 1: T , w 1: T , ˆ H 1: T ) ( x 1: T | T , Σ 1: T | T ) =

Original: Smoothing Step ( x +1: T , Σ +1: T , J 1: T , ˆ F 1: T )
Revised: Smoothing Step ( x +1: T , Σ +1: T , J 1: T , ˆ F 1: T )

Original: o +1:
Revised: o +1:

Original: T = decoder ( x 1: T | T , Σ 1: T | T )
Revised: T = decoder ( x 1: T | T , Σ 1: T | T )

Original: L 1: T = - Likelihood ( gt 1: T , o +1: T ) Backward Propagation ()
Revised: L 1: T = - Likelihood ( gt 1: T , o +1: T ) Backward Propagation ()

Original: [Table]
Revised: [Table]

Original: Input: Ground Truth gt 1: T , Observations y 1: T , last posteriors ( x +1: T , Σ
Revised: Input: Ground Truth gt 1: T , Observations y 1: T , last posteriors ( x +1: T , Σ

Original: +1: T ) , initial posterior ( x +0 , Σ +0 ) if Dynamics are not known then
Revised: +1: T ) , initial posterior ( x +0 , Σ +0 ) if Dynamics are not known then

Original: α 1: T = Dynamics Network ( x +0: T − 1 )
Revised: α 1: T = Dynamics Network ( x +0: T − 1 )

Original: Obtain ˆ F 1: T and ˆ H 1: T by (12) ( w 1: T , r 1: T ) =
Revised: Obtain ˆ F 1: T and ˆ H 1: T by (13) ( w 1: T , r 1: T ) =

Original: MLP ( y 1: T ) ( x − 1: T , Σ − 1: T ) =
Revised: MLP ( y 1: T ) ( x − 1: T , Σ − 1: T ) =

Original: Prediction Step ( x +0: T − 1 ,
Revised: Prediction Step ( x +0: T − 1 ,

Original: Σ +0: T − 1 , ˆ F 1: T )
Revised: Σ +0: T − 1 , ˆ F 1: T )

Original: K 1: T = Σ − 1: T
Revised: K 1: T = Σ − 1: T

Original: ˆ H 1: T M 1: T M T 1: T ,
Revised: ˆ H 1: T M 1: T M T 1: T ,

Original: M 1: T =
Revised: M 1: T =

Original: GRU KG ( Σ − 1: T , r 1: T )
Revised: GRU KG ( Σ − 1: T , r 1: T )

Original: J 1: T = Σ +1:
Revised: J 1: T = Σ +1:

Original: T ˆ F T 1: T N 1: T N T 1: T ,
Revised: T ˆ F T 1: T N 1: T N T 1: T ,

Original: N 1: T =
Revised: N 1: T =

Original: GRU SG ( Σ − 1: T ) ( x +1: T , Σ +1: T ) =
Revised: GRU SG ( Σ − 1: T ) ( x +1: T , Σ +1: T ) =

Original: Filtering Step ( x − 1: T , Σ − 1: T , K 1: T , w 1: T , ˆ H 1: T ) ( x 1: T | T , Σ 1: T | T ) =
Revised: Filtering Step ( x − 1: T , Σ − 1: T , K 1: T , w 1: T , ˆ H 1: T ) ( x 1: T | T , Σ 1: T | T ) =

Original: Smoothing Step ( x +1: T , Σ +1: T , J 1: T , ˆ F 1: T )
Revised: Smoothing Step ( x +1: T , Σ +1: T , J 1: T , ˆ F 1: T )

Original: o +1:
Revised: o +1:

Original: T =
Revised: T =

Original: MLP ( x 1: T | T + , Σ 1: T | T )
Revised: MLP ( x 1: T | T + , Σ 1: T | T )

Original: L 1: T = - Likelihood ( gt 1: T , o +1: T ) Backward Propagation () else
Revised: L 1: T = - Likelihood ( gt 1: T , o +1: T ) Backward Propagation () else

Original: Q network =
Revised: Q network =

Original: MLP ( x + 0: T − 1 ) or GRU ( x + 0: T − 1 , Q 1: T ) ( F 1: T , H 1: T ) =
Revised: MLP ( x + 0: T − 1 ) or GRU ( x + 0: T − 1 , Q 1: T ) ( F 1: T , H 1: T ) =

Original: Dynamics r 1: T = trainable layer ( y 1: T ) q 1: T = Q network ( x +0: T − 1 ) ( x − 1: T , Σ − 1: T ) =
Revised: Dynamics r 1: T = trainable layer ( y 1: T ) q 1: T = Q network ( x +0: T − 1 ) ( x − 1: T , Σ − 1: T ) =

Original: Prediction Step (( x +0: T − 1 , Σ +0: T − 1 ) , Q 1: T , F 1: T )
Revised: Prediction Step (( x +0: T − 1 , Σ +0: T − 1 ) , Q 1: T , F 1: T )

Original: K 1: T = Σ − 1: T H 1: T M 1: T M T 1: T ,
Revised: K 1: T = Σ − 1: T H 1: T M 1: T M T 1: T ,

Original: M 1: T =
Revised: M 1: T =

Original: GRU KG ( Σ − 1: T , r 1: T )
Revised: GRU KG ( Σ − 1: T , r 1: T )

Original: J 1: T = Σ +1: T F T 1: T N 1: T N T 1: T ,
Revised: J 1: T = Σ +1: T F T 1: T N 1: T N T 1: T ,

Original: N 1: T =
Revised: N 1: T =

Original: GRU SG ( Σ − 1: T ) ( x +1: T , Σ +1: T ) =
Revised: GRU SG ( Σ − 1: T ) ( x +1: T , Σ +1: T ) =

Original: Filtering Step ( x − 1: T , Σ − 1: T , K 1: T , y 1: T , H 1: T ) ( x 1: T | T , Σ 1: T | T ) =
Revised: Filtering Step ( x − 1: T , Σ − 1: T , K 1: T , y 1: T , H 1: T ) ( x 1: T | T , Σ 1: T | T ) =

Original: Smoothing Step ( x +1: T , Σ +1: T , J 1: T , F 1: T ) σ 1: T | T =
Revised: Smoothing Step ( x +1: T , Σ +1: T , J 1: T , F 1: T ) σ 1: T | T =

Original: Trainable Layer ( Σ 1: T | T )
Revised: Trainable Layer ( Σ 1: T | T )

Original: o +1:
Revised: o +1:

Original: T =
Revised: T =

Original: [ x 1: T | T , σ 1: T | T ]
Revised: [ x 1: T | T , σ 1: T | T ]

Original: L 1: T = - Likelihood ( gt 1: T , o +1: T ) Backward Propagation () end if
Revised: L 1: T = - Likelihood ( gt 1: T , o +1: T ) Backward Propagation () end if

Original: A. P YTHON INFERENCE CODE
Revised: A. P YTHON INFERENCE CODE

Original: To demonstrate the simplicity of our proposed GIN, we include intuitive inference code with Tensorflow library for both the high dimensional and low dimensional experiments.
Revised: To demonstrate the simplicity of our proposed GIN, we include intuitive inference code with Tensorflow library for both the high dimensional and low dimensional experiments.

Original: The code runs with
Revised: The code runs with

Original: Python 3.6+.
Revised: Python 3.6+.

Original: The entire code to reproduce the experiments are available in Github repository.
Revised: The entire code to reproduce the experiments are available in Github repository.

Original: A.12. P YTHON INTUITIVE CODE FOR HIGH DIMENSIONAL EXPERIMENTS .
Revised: A.12. P YTHON INTUITIVE CODE FOR HIGH DIMENSIONAL EXPERIMENTS .

Original:  initial_states 
Revised: 27 28 class 34def call 42 o initial_states self.

Original:  = Dynamics self.
Revised: 1:T = Dynamics self.

Original: GIN_CELL_OBJ = self.
Revised: GIN_CELL_OBJ = self.

Original: _CELL x{+GIN_CELL .x_
+}
Revised: {+GIN_CELL .x_
+}

Original: Sigma
Revised: Sigma_0_

Original: ) self.
Revised: ) self.

Original: MLP_ENC  MLP_ENC _DEC
Revised: MLP_ENC = MLP_ENC self.

Original: _DEC 34def call o 1:T):
Revised: MLP_DEC = MLP_DEC 34def call( o_1:T): r

Original: ENC o
Revised: _ENC(o_

Original: :T) x
Revised: :T) x

Revised: CELL_ o

Original: Sigma
Revised: Sigma

Original: return
Revised: return

Original: 42 43 Data = DataGen() 44 Dynamics_
Revised: 42 43 Data = DataGen() 44 Dynamics_Matrices = () 45 x

Original: Matrices)
Revised: _Matrices)