Original: R ECURRENT N EURAL N ETWORK A RCHITECTURE BASED ON D YNAMIC S YSTEMS T HEORY FOR D ATA D RIVEN M ODELLING OF C OMPLEX P HYSICAL S YSTEMS
Revised: R ECURRENT N EURAL N ETWORK A RCHITECTURE BASED ON D YNAMIC S YSTEMS T HEORY FOR D ATA D RIVEN MODELLING OF C OMPLEX P HYSICAL S YSTEMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While dynamic systems can be modelled as sequence-to-sequence tasks by deep learning using different network architectures like DNN, CNN, RNNs or neural
Revised: While dynamic systems can be modeled as sequence-to-sequence tasks by deep learning using different network architectures like DNN, CNN, RNNs or neural

Original: ODEs, the resulting models often provide poor understanding of the underlying system properties.
Revised: ODEs, the resulting models often provide poor understanding of the underlying system properties.

Original: We propose a new recurrent network architecture, the Dynamic
Revised: We propose a new recurrent network architecture, the Dynamic

Original: Recurrent Network (DYRNN), where the computation function is based on the discrete difference equations of basic linear system transfer functions known from dynamic system identiﬁcation.
Revised: Recurrent Network (DYRNN), where the computation function is based on the discrete difference equations of basic linear system transfer functions known from dynamic system identiﬁcation.

Original: This results in a more explainable model, since the learnt weights can provide insight on a system’s time dependent behaviour.
Revised: This results in a more explainable model, since the learnt weights can provide insight on a system’s time dependent behaviour.

Original: It also introduces the sequences’ sampling rate as an additional model parameter, which can be leveraged, for example, for time series data augmentation and model robustness checks.
Revised: It also introduces the sequences’ sampling rate as an additional model parameter, which can be leveraged, for example, for time series data augmentation and model robustness checks.

Original: The network is trained using traditional gradient descent optimization and can be used in combination with other state of the art neural network layers.
Revised: The network is trained using traditional gradient descent optimization and can be used in combination with other state of the art neural network layers.

Original: We show that our new layer type yieldsbetter results  than other recurrent layer types on several system identiﬁcation tasks.
Revised: We show that our new layer type yields results comparable to or better than other recurrent layer types on several system identiﬁcation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dynamic systems occur in many different areas of life (Isermann & M ¨ unchhof (2011)).
Revised: Dynamic systems occur in many different areas of life (Isermann & M ¨ unchhof (2011)).

Original: From biology, engineering, medicine to economics and more: Often, if a system changes its state based on a external input, this system can be viewed as a dynamic system.
Revised: From biology, engineering, medicine to economics and more: Often, if a system changes its state based on a external input, this system can be viewed as a dynamic system.

Original: Dynamic system identiﬁcation is the process of modelling the system’s properties.
Revised: Dynamic system identiﬁcation is the process of modelling the system’s properties.

Original: Such models can be used, for example, for anomaly detection, controller design or outcome prediction.
Revised: Such models can be used, for example, for anomaly detection, controller design or outcome prediction.

Original: For linear systems, this identiﬁcation task is already well understood and state of the art methods exist.
Revised: For linear systems, this identiﬁcation task is already well understood and state of the art methods exist.

Original: However, if a system exhibits non-linear behaviour, for example slip-stick-effects due to mechanical friction, the applicability of these methods is limited.
Revised: However, if a system exhibits non-linear behaviour, for example slip-stick-effects due to mechanical friction, the applicability of these methods is limited.

Original: In this case different approaches implemented in the state of the art range from white-box to black-box models.
Revised: In this case different approaches implemented in the state of the art range from white-box to black-box models.

Original: Generally, increasing system complexity raises the need for more powerful and often less understandable model architectures in order to produce satisfactory results: White box (based on differential equations or numerical simulations of the physical system components), black box systems (like Gaussian processes, deep neural networks, Support Vector Machines) and grey box models, which often employ a mix of linear and non-linear building blocks.
Revised: Generally, increasing system complexity raises the need for more powerful and often less understandable model architectures in order to produce satisfactory results: White box (based on differential equations or numerical simulations of the physical system components), black box systems (like Gaussian processes, deep neural networks, Support Vector Machines) and grey box models, which often employ a mix of linear and non-linear building blocks.

Original: One example of a tool used in engineering are HammersteinWiener models which are a combination of linear and (prior known) non-linear equations (shown in Figure 1).
Revised: One example of a tool used in engineering are Hammerstein-Wiener models which are a combination of linear and (prior known) non-linear equations (shown in Figure 1).

Original: The linear model parameters are determined based on the training data.
Revised: The linear model parameters are determined based on the training data.

Original: The non-linear behaviour of models is modelled using lookup tables or user deﬁned non-linear functions.
Revised: The non-linear behaviour of models is modeled using lookup tables or user deﬁned non-linear functions.

Original: In this work we present a new type of recurrent neural network layer called the Dynamic Recurrent  Network (DYRNN).
Revised: In this work we present a new type of recurrent neural network layer called the Dynamic Recurrent Neural Network (DYRNN).

Original: It is designed for data based modelling of dynamic systems in a sequence-tosequence manner based on input ( x ( t ) ) and output ( y ( t ) ) data.
Revised: It is designed for data based modelling of dynamic systems in a sequence-to-sequence manner based on input ( x ( t ) ) and output ( y ( t ) ) data.

Original: With it, we intend to bridge the gap between dynamic systems theory and recurrent neural networks.
Revised: With it, we intend to bridge the gap between dynamic systems theory and recurrent neural networks.

Original: The layer’s internal computation is based on elemental transfer blocks from linear system identiﬁcation.
Revised: The layer’s internal computation is based on elemental transfer blocks from linear system identiﬁcation.

Revised: By combining it with non-linear neural networks, a Hammerstein-Wiener style model is emulated.

Original: This way, the model can offer background knowledge about the examined system’s internal properties.
Revised: This way, the model can offer additional knowledge about the examined system’s internal properties.

Original: Additionally, while the model is trained on sampled data of one sampling rate it can be applied to data of the same system at a different sampling rate.
Revised: Furthermore, while the model is trained on sampled data of one sampling rate it can be applied to data of the same system at a different sampling rate.

Original: This can be used to check the robustness of the model or to save time during training.
Revised: This can be used to check the robustness of the model or to save time during training.

Original: We show that our network produces results which are better than  other recurrent networks (RNN, LSTM, GRU) on three different problem datasets.
Revised: We show that our network produces results which are better than or comparable to other recurrent networks (RNN, LSTM, GRU) on three different problem datasets.

Original: Since the layer can be implemented to be compatible to current deep learning frameworks, it can be combined with state of the art neural network layers (like convolutional or fully connected layers) and training techniques.
Revised: Since the layer can be implemented to be compatible to current deep learning frameworks, it can be combined with state of the art neural network layers (like convolutional or fully connected layers) and training techniques.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Dynamic system identiﬁcation can be viewed as a sequence-to-sequence task of the modelling of a systems’ outputmeasurements based on certain input measurements.
Revised: Dynamic system identiﬁcation can be viewed as a sequence-to-sequence task of the modelling of a systems’ output based on certain inputs.

Original: Isermann & M ¨ unchhof (2011), for example, list several different tools like ARIMA processes for linear systems and multiple neural network architectures for non-linear systems.
Revised: Isermann & M ¨ unchhof (2011), for example, list several different tools like ARIMA processes for linear systems and multiple neural network architectures for non-linear systems.

Original: Examples for the latter are locally recurrent locally feedforward networks (LRGF), Multi Layer Perceptrons (MLP) and Radial Basis Function (RBF) networks of different types of dynamics.
Revised: Examples for the latter are locally recurrent locally feedforward networks (LRGF), Multi Layer Perceptrons (MLP) and Radial Basis Function (RBF) networks of different types of dynamics.

Original: These model structures are generalized, however, and as we will show, further theoretical background on linear systems theory could be leveraged in the construction of these models.
Revised: These model structures are generalized, however, and as we will show, further theoretical background on linear systems theory could be leveraged.

Original: Generally, Deep learning offers multiple neural network layer types that can be employed when dealing with sequence-to-sequence problems, like fully connected (FC) networks, convolutional networks (CNN) or recurrent networks.
Revised: Generally, deep learning offers multiple neural network layer types that can be employed when dealing with sequence-to-sequence problems, like fully connected (FC) networks, convolutional networks (CNN) or recurrent networks.

Original: The latter are also known as sequential models (like RNN, LSTM by Hochreiter & Schmidhuber (1997) and GRU by Cho et al.
Revised: Recurrent networks are also known as sequential models (like RNN, LSTM by Hochreiter & Schmidhuber (1997) and GRU by Cho et al.

Original: and are state of the art layer types for text based sequence-to-sequence problems like machine translation or text processing.
Revised: and have been used successfully for text based sequence-to-sequence problems like machine translation or text processing.

Original: Wang (2017) demonstrates a concept of LSTM for dynamic system identiﬁcation by using several parallel LSTM layers which predict the systems behaviour based on its input and prior predictions and their derivatives.
Revised: Wang (2017) demonstrates a concept of LSTM for dynamic system identiﬁcation by using several parallel LSTM layers which predict the systems behaviour based on its input and prior predictions and their derivatives.

Original: A different approach of modelling dynamic systems are neural ordinary differential equations (ODEs) by Chen et al.
Revised: A different approach of modelling dynamic systems are neural ordinary differential equations (ODEs) by Chen et al.

Original: These networks learn dy/dt of a function f with y ( t ) =
Revised: These networks learn dy/dt of a function f with y ( t ) =

Original: f ( x ( t )) and the resulting ODE model is used an numerical integrator/solver (like Runge-Kutta) to compute y ( t ) .
Revised: f ( x ( t )) and the resulting ODE model is used an numerical integrator/solver (like Runge-Kutta) to compute y ( t ) .

Original: This has the advantage of a varying sampling step size which is determined by the solver, but these methods are agnostic of dynamic systems theory knowledge.
Revised: This has the advantage of a varying sampling step size which is determined by the solver, but these methods are agnostic of dynamic systems theory knowledge.

Original: Similarly, Raissi et al.
Revised: Similarly, Raissi et al.

Original: (2019) use deep learning to learn Partial Differential Equations (PDE) of physical systems in a FC model combined with a numerical integrator.
Revised: (2019) use deep learning to learn Partial Differential Equations (PDE) of physical systems in a FC model combined with a numerical integrator.

Original: Furthermore, since the evaluation of ODE/PDE models is done using a numerical integrator, the model is difﬁcult to apply in combination with other neural network layers like for example convolutional or recurrent layers.
Revised: Furthermore, since the evaluation of ODE/PDE models is done using a numerical integrator, the model is difﬁcult to apply in combination with other neural network layers like for example convolutional or recurrent layers.

Original: In terms of sampling frequency of the measurement data, recurrent network architectures can only be trained on one speciﬁc data frequency, and do not provide the functionality to generalize to other sampling rates of the same system.
Revised: In terms of sampling frequency of the measurement data, recurrent network architectures can only be trained on one speciﬁc data frequency, and do not provide the functionality to generalize to other sampling rates of the same system.

Original: In such a case one would have to resample the new data to the frequency of the training set.
Revised: In such a case one would have to resample the new data to the frequency of the training set.

Original: Explainability approaches for sequential models in text processing deduce which parts of the sentences are relevant for the model’s prediction based on the activation of the internal gates (as shown by e.g.Krakovna & Doshi-Velez (2016)).
Revised: Explainability approaches for sequential models in text processing deduce which parts of the sentences are relevant for the model’s prediction based on the activation of the internal gates (as shown by e.g.Krakovna & Doshi-Velez (2016)).

Original: Interpretability of RNN/LSTM or GRU models for continuous measurement data has not been explored yet to our knowledge.
Revised: Interpretability of RNN/LSTM or GRU models for continuous measurement data has not been explored yet to our knowledge.

Original:  D YNAMIC R ECURRENT N ETWORK
Revised:  D YNAMIC R ECURRENT N ETWORK

Revised: The complexity of the modelling of dynamic systems does not only result from the potential nonlinearity, but also from the fact that the model has to keep track of the system’s current and past states in order to predict the output based on new input.

Original: In the following, we ﬁrst explain the theoretical background from dynamic systems theory used in this work.
Revised: We intend to model a dynamic system in [Figure] a sequence-to-sequence task, which is why we chose a recurrent network architecture.

Revised: Recurrent neural networks can be seen as units which iterate over a given sequence and predict an output.

Revised: During this computation, a hidden state is computed which is leveraged for the prediction of the next time step.

Revised: Our network differs from other state of the art recurrent networks in its computation function, which is derived from linear dynamic systems theory.

Revised: In the following we explain the theoretical background used in this work.

Original: Then, we describe the structure of our DYRNN.
Revised: Then we describe the structure of our DYRNN.

Original: Finally, we show different advantages that result from this structure like training and prediction at different signal sampling rates and interpretability of the models.
Revised: Finally, we show different advantages that result from this structure like training and prediction at different signal sampling rates and interpretability of the models.

Original: 3. L AYER D ESIGN
Revised: 3. L AYER S TRUCTURE

Original: Our layer’s computation function is derived from dynamic system’s theory.

Original: The background knowledge in this section is documented by F¨ollinger et al.
Revised: The background knowledge in this section is covered by F ¨ ollinger et al.

Original: (2016) and Yarlagadda (2010).
Revised: (2016) and Yarlagadda (2010).

Original: In dynamic systems theory a linear system with the external input u ( t ) and resulting system output y ( t ) is expressed as the differential equation
Revised: In dynamic systems theory a linear system with the external input u ( t ) and resulting system output y ( t ) is expressed as the differential equation

Original: [Equation]
Revised: [Equation]

Original: Therefore, a linear system acts as transformation of the input u ( t ) based to a convolution ( ∗ ) with the system’s linear transfer function g ( t )(shown in e.g. Yarlagadda (2010)) to produce the system output y ( t ) with
Revised: Therefore, a linear system acts as transformation of the input u ( t ) based to a convolution ( ∗ ) with the system’s linear transfer function g ( t ) to produce the system output y ( t ) with

Original: [Equation]
Revised: [Equation]

Original: The transfer function g ( t ) is used in engineering for, amongst others, controller design, system stability analysis or frequency response estimation.
Revised: The transfer function g ( t ) is used in engineering for, amongst others, controller design, system stability analysis or frequency response estimation.

Original: Larger, more complicated transfer functions can be seen as several basic functions which are interconnected in a circuit type matter in parallel or in series (see Figure 2). [Figure]
Revised: Larger, more complicated transfer functions can be seen as several basic functions which are interconnected in a circuit type fashion in parallel or in series (see Figure 3).

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original:  Dynamic systems theory identiﬁes ﬁve basic linear transfer functions from which all linear dynamic systems can be modelled, which we use in the construction of the DYRNN:
Revised: [Equation] Dynamic systems theory identiﬁes ﬁve basic linear transfer functions from which all linear dynamic systems can be modeled, which we use in the construction of the DYRNN:

Original: P, I, D, PT1 and PD (see also Appendix Section A).
Revised: P, I, D, PT1 and PD. For further information and and visualization see Appendix Section A).

Original: They have the following functionalities:
Revised: They have the following functionalities:

Original: • P: P roportional gain of the input, implemented as a multiplication with a constant • I: I ntegrating component, meaning the step-wise summation of an input over time. 
Revised: • P: P roportional gain of the input, implemented as a multiplication with a constant • I: I ntegrating component, meaning the step-wise summation of an input over time. • D: D ifferential component acting as a high-pass ﬁlter [Equation]

Original: • D: D ifferential component acting as a high-pass ﬁlter • PT1: P roportional multiplication of input with t ime delay, used to model e
Revised: [Figure] [Equation] [Figure] • PT1: P roportional multiplication of input with t ime delay, used to model e

Original: g. voltage in capacitors in RC circuits.
Revised: g. voltage in capacitors in RC circuits.

Original: This function also acts as a low-pass ﬁlter • PD: P roportional increase with a d ifferential component
Revised: This function also acts as a low-pass ﬁlter • PD: P roportional increase with a d ifferential component

Original: In the following equations K stands for a constant with inﬂuence on the output amplitude, while T is a time constant which inﬂuences the speed of the system’s reaction. 
Revised: In the following equations K stands for a constant with inﬂuence on the output amplitude, while T is a time constant which inﬂuences the speed of the system’s reaction. K and T act as the trainable weights in a DYRNN layer.

Original: For usage in a recurrent layer, these differential equations are discretized with – in our case – ﬁrst degree forward Euler.
Revised: For usage in a recurrent layer, these differential equations are discretized with – in our case – ﬁrst degree forward Euler.

Original: This is common in many engineering applications, and means replacing ˙ y ( t ) with
Revised: This is common in many engineering applications, and means replacing ˙ y ( t ) with

Original: [Equation]
Revised: [Equation]

Original: This results in discrete recurrence equations, with the sample number k and the time distance between samples ∆ t .
Revised: This results in discrete recurrence equations, with the sample number k and the time distance between samples ∆ t .

Original: The respective equations of the basic transfer functions are as follows:
Revised: The respective equations of the basic transfer functions are as follows:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] [Figure] x(k)
Revised: with the input x ( k ) and all K, T > 0 .

Original: P I D
Revised: The equations above are implemented as the computation function of a recurrent network layer as described in Appendix Section A.1.

Original: PT PD p(k) i(k) d(k) pt1(k) pd(k) i(k-1) x(k-1) pt1(k-1) x(k-1) [Figure] with all K, T > 0 .
Revised: K and T in the equations become trainable weights of the layer, while the hidden state consists of x ( k − 1) , i ( k − 1) and pt ( k − 1) .

Original: The equations above are implemented as the computation function of a recurrent network layer as described in Appendix Section A.1.
Revised: In the following we refer to these internal computations as subcomponents.

Original: Recurrent models incorporate a unit which iterates over the layer’s input sequence while keeping additional data in a hidden state to provide information necessary for the computation of the next step’s result (see Figure 3a).
Revised: We explore two different network variants in our work: The DYRNN5 with all ﬁve known subcomponents and the DYRNN3 with just P, PD and PT1. The reason for this is, that a D subcomponent can be approximated by PD and I by a PT. Since integrators can also cause model instabilites, we explore both variants in our experiments.

Original: We formulate one unit’s computations based on the number of input channels n ic and the number of outputs per basic transfer function type n oc .
Revised: We formulate one unit’s computations based on the number of input channels n ic and the number of outputs per subcomponent type n oc .

Original: The number of output channels per layer n layer amounts to (base component count * n oc = 5 *{+
+} n oc ), as shown in Figure 3b.
Revised: The number of output channels per layer n layer amounts to (subcomponent count) * n oc = {+
+} n oc for DYRNN3 and 5* n oc for DYRNN5 (see Figure 4a).

Original: The fact that DYRNN can be implemented with an interface compatible with other recurrent networks allows the modelling of a systems properties by training in a sequence-to-sequence fashion using out of the box backpropagation through time, gradient descent and optimizer functions as implemented in state of the art frameworks like Pytorch (by Paszke et al.
Revised: The fact that DYRNN can be implemented with an interface compatible with other recurrent networks allows the modelling of a systems properties by training in a sequence-to-sequence fashion using out of the box backpropagation through time, gradient descent and optimizer functions as implemented in state of the art frameworks like Pytorch (by Paszke et al.

Original:  and Tensorﬂow (by Mart \ ’ıın Abadi et al.
Revised: (2019)) and Tensorﬂow (by Mart´ın Abadi et al.

Original: In the following we present an example architecture using DYRNN and several different features that can be leveraged due from this design.

Original: 3. E XAMPLE A RCHITECTURE

Original: Since each input of the layer is connected to ﬁve basic components, stacking DYRNN layers results in an increasing number of output channels, as shown in Figure 4.
Revised: Since each input of the layer is connected to ﬁve basic components, stacking DYRNN layers results in an increasing number of output channels, as shown in Figure 4b.

Original: The architecture we used in our experiments is a cascadingrow of DYRNN layers, which means that all inputs and intermediate results are concatenated and passed to a per-time-step FC network as shown in Figure 5.
Revised: In our experiments, we achieved good results with two cascading DYRNN layers followed by a per-time step linear FC layer with one neuron.

Original: Using nonlinear activation functions, this design results in a similar structure as known from HammersteinWiener models for modelling of static non-linearities in the data.
Revised: This ﬁnal layer can be enhanced using non-linear activation functions and more layers/neurons, which would result in a similar structure to Hammerstein-Wiener models for modelling of static non-linearities in the data.

Original: In case of noisy input data, for example due to sensor noise, additional convolutional layers could be used as well.[Equation] [Figure] (e
Revised: In case of noisy input data, for example due to sensor noise, additional CNN layers could be used as well.

Original: [Figure]

Original: g. Fully

Original: [Figure]

Original: y(k) =

Original: DYRNN Layer

Original: 3. G ENERALIZATION FOR D IFFERENT S IGNAL S AMPLING R ATES
Revised: 3. G ENERALIZATION FOR D IFFERENT S IGNAL S AMPLING R ATES

Original: The new parameter ∆ t means that a model can be used on datasets with a different sampling rates by adjusting the ∆ t when interacting with the model.
Revised: The new parameter ∆ t means that a model can be used on datasets with a different sampling rates by adjusting the ∆ t when interacting with the model.

Original: Resampling can be leveraged as a new means to perform time series data augmentation.
Revised: This can be leveraged as a new means to perform time series data augmentation.

Original: Training on different rates and testing on another sampling rate can also provide insight about a model’s robustness.
Revised: Training on different rates and testing on another sampling rate can also provide insight about a model’s robustness.

Original: Additionally, training time can be reduced by subsampling the dataset during training [Figure] and using the original sampling rate while inferencing for greater prediction accuracy (see Figure 6)
Revised: Additionally, training time can be reduced by subsampling the dataset during training, and using the original sampling rate while inferencing for greater prediction accuracy (see Figure 5).

Original: [Figure]
Revised: [Figure]

Original: 3. I NTERPRETABILITY OF R ESULTING M ODELS
Revised: 3. I NTERPRETABILITY OF R ESULTING M ODELS

Original: Based on architectures like in Section 3.2, it is possible to interpret the signiﬁcance of a variable towards the ﬁnal dataset as well as the properties of its inﬂuence.
Revised: Based on our layer, it is possible to interpret the signiﬁcance of a subcomponent towards the ﬁnal dataset as well as the properties of its inﬂuence.

Original: While the DYRNN layers learn the time dependant system behaviour, a per-timestep fully connected network selects the relevant dynamics of the model.
Revised: While the DYRNN layers learn the time dependant system behaviour, a per-timestep fully connected network selects the relevant dynamics of the model.

Original: The higher the FC’s learnt weight connected to a speciﬁc subunit, the more signiﬁcant it is for the modelling of the dynamic system (see Figure 7).
Revised: The higher the FC’s learnt weight connected to a speciﬁc subunit, the more signiﬁcant it is for the modelling of the dynamic system (see Figure 6).

Original: The FC layer’s weights are used globally and in a time independent manner. Thereby the edge weight can be interpreted as the global importance of a DYRNN sub-computation in the modelling of the system.
Revised: As an example, assume a network of one

Original:  Figure 7 visualizes this.
Revised: DYRNN5 layer and a fully connected (FC) layer as shown in Figure 6.

Original: For example, given a network of one DYRNN layer and a fully connected layer, if the subunit P, I and PT1 are weightedby the FC with 0.5, 1.5 and 0.8 as opposed to the D and PD with 0.1 and 0.2,this shows that the modelled system mostly displays P, I and PT1 properties. 
Revised: Since the subcomponents P, I and PT1 are weighted with 0.5, 1.5 and 0.8 as opposed to the D and PD with 0.1 and 0.2, the modeled system mostly displays P, I and PT1 properties. Additionally, we can analyse the model by evaluating its equation in the Laplace and the Fourier domain.

Original: This architecture means that we do not rely upon the DYRNN’s hidden states for interpretation.

Original: Instead, the static part of the network is designed to select relevant system properties, while enabling the modelling of non-linear parts of the system. Nonetheless interpreting the results will become more difﬁcult if complexity of the static part of the  network increases. Further methods like L1/L2 regularization in the FC layers can also be of use in this process.
Revised: This means replacing the discrete subcomponents with their respective Laplace formulas as described in the Appendix Section A, while keeping the structure of the network as well as the trained values for K and T. This yields a new equation depending on the parameter s , which for the example network above results in:

Original: [Figure]
Revised: [Equation]

Original: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: u

Original: [Figure]
Revised: [Equation]

Original:  ) y(
Revised: k ) y

Original: [Figure]
Revised: [Equation]

Original: [Figure] [Figure] [Figure]
Revised: y i ( k ) y d ( k ) y pt 1 ( k ) y pd ( k ) y ( k ) 0.

Original: DYRNN
Revised: DYRNN

Original: FC
Revised: FC

Original: E XPERIMENTS
Revised: The Laplace form has the advantage that a convolution with the transfer function y ( t ) = u ( t ) ∗ g ( t ) can be replaced with a multiplication with Y ( s ) = G ( s ) · U ( s ) .

Original: We evaluated our network on three different datasets.
Revised: It allows an engineer to analyse the model for stability, and to visualize different kinds of information on the model, e.g. in a polezero plot. The transfer function G ( s ) can be examined for system stability by replacing s with iω ( ω = 2 π ∗ f , with f as the signal frequency) yields the Fourier from of the transfer function. This can be used for frequency response and root locus analysis (F¨ollinger et al. (2016)), as we show in our experiments.

Original: The ﬁrst is an electical circuit simulated using using Matlab Simulink, as shown in the Appendix in Figure 12. The input of  our model is the input voltage of the circuit, while the output value is the voltage measured over the capacitor.
Revised: The transformation of G ( s ) for our example network into the time domain yields the following differential equation: [Equation] Transforming larger networks – for example the one used in our experiments – works along the same lines.

Original: The simulation’s sampling rate is 0.005s.
Revised: Such extracted differential equations can further be leveraged to analyze the model.

Original: We simulated three time series for training, validation and testing with length of 2 seconds.
Revised: We evaluated our network on three different datasets.

Original: The input consists of a square signal with randomly changing amplitudes from 0.5s to 1.5s.
Revised: The ﬁrst is an electrical RC-circuit simulated using Matlab Simulink, as shown in the Appendix in Figure 13.

Original: Theremainder of the time, the input voltage is 0V. The other two datasets are from DaISy (De Moor B.L.R. (a) and  De Moor B.L.R. (b)).
Revised: The other two datasets are from the Database for Identiﬁcation of Systems ”DaISy”, namely the Heating System De Moor B.L.R. (b) and the Thermic Resistance De Moor B.L.R. (a)) datasets.

Original: Further information about the  experiments is documented in Appendix Section A.2.
Revised: In the following we will refer to the datasets as Circuit, Heating and Thermic. In these experiments we focus on learning the correct dynamics independent from the initial value of y(t). Therefore, we subtract the mean time offset of the prediction sequence towards the label before the ﬁnal evaluation with Mean Squared Error.

Original: [Figure]
Revised: [Equation]

Original: The ﬁrst goal of our experiments is to show that our model performs well compared toother recurrent networks, namely RNN, LSTM and GRU as implemented in Tensorﬂow.
Revised: The ﬁrst goal of the experiments is to evaluate our models (DYRNN3 and DYRNN5) compared to RNN, LSTM and GRU as implemented in Tensorﬂow.

Original: The network architecture isas shown in Figure 8.
Revised: The network architecture is shown in Figure 7.

Original: The recurrent part of the network is built in a cascaded fashion of two recurrent layers which are concatenated with the original input.
Revised: The ﬁrst part of the network is built in a cascaded fashion of two recurrent layers which are concatenated with the original input.

Original: Its results are passed to a linear fully-connected layer with one unit.
Revised: Its results are passed to a linear fully-connected layer with one unit.

Original: The number of units in the  recurrent layers were chosen to result in a similar parameter distribution to DYRNN, as shown in Table 1 for the ﬁrst dataset (Appendix
Revised: The number of units in the other recurrent layers were chosen to result in a similar parameter distribution to DYRNN5, as shown in Section A.2.

Original:  Table 5 and 6). 
Revised: For experiment a similar count of parameters in the recurrent part of the network and the same layer constellations were used (see Table 4,5 and 6). Due to the design of the DYRNN5 network, its FC part contains more parameters than the other networks.

Original: Additionally, for the Circuit dataset we train the model on different sampling rates and examine its ability to generalize to other sampling rates.
Revised: Additionally, a DYRNN5 model was trained for different sampling rates for the Circuit dataset to examine its ability to generalize to other sampling rates.

Original: For this,we subsampled the dataset  at different frequencies using linear interpolation.
Revised: For this, the dataset was subsampled at different frequencies using linear interpolation.

Original: The constellation of sampling rates in these three runs are described in Table 2.
Revised: The constellation of sampling rates in these three runs are described in Table 1.

Original: All networks were trained with the Adam Optimizer by Kingma & Ba (2014) and the Huber Loss function with standard parameters deﬁned in Tensorﬂow.
Revised: All networks were trained with the Adam Optimizer by Kingma & Ba (2014) and the Huber Loss function with standard parameters deﬁned in Tensorﬂow.

Original: The amount of training iterations waskept at a maximum of 5000 epochs for Circuit and 3000 for the
Revised: The amount of training iterations was 3000 epochs. Evaluation of results is done on a separate testing dataset.

Original: DaISy datasets.
Revised: [Table]

Original: For Circuit, trainingwas terminated once overﬁtting was detected compared to the validation set. Evaluation of results is done on a seperate testing dataset.
Revised: After training the DYRNN models are transformed into Laplace transfer functions G(s) using Sympy (Meurer et al. and three different views of this function are generated: G(s) over s, the frequency response | G ( iω ) | over ω and the root locus curve of G ( iω ) over ω to compare the different model dynamics per dataset.

Original:  R ESULTS
Revised:  R ESULTS

Original: After 10 runs each, the different recurrent networks performance on the testing set is shown inFigure 10.
Revised: After 10 runs each, the different recurrent networks’ performance on the testing set is shown in

Original: Figure 9 shows the best and worst runs on the testing set for the Circuit dataset.
Revised: Figure 8. Since an activation function in the FC layer was omitted, it amounts to a linear combination of all channels from the cascaded part. Therefore, it can be assumed that a better performance of the DYRNN stems from its computation functions.

Original: On this dataset,
Revised: In this case a different network conﬁguration and more parameters may enable the other layer types to achieve better results. Section A.3 in the Appendix shows the best and worst runs on the testing set for all datasets and layer types as well as the transfer functions learnt by DYRNN3 rounded to two signiﬁcant digits in Section A.4. The transfer function analysis plots in Figures 10, 11 and 12 were generated for the runs which performed best on the testing datasets.

Original: DYRNN performs consistently better than each of the  recurrent networks.
Revised: On the Circuit dataset, both DYRNN5 and DYRNN3 perform consistently better than each of the other recurrent networks.

Original: This is to be expected since an idealized capacitor acts as a PT1 element, so the DYRNN is predestined for this modelling task.
Revised: This is to be expected since an idealized capacitor acts as a PT1 element, so the DYRNN is predestined for this modelling task.

Original: Nonetheless, all layer types preform well on the dataset, if training was successful.
Revised: Nonetheless, all layer types perform well on the dataset if training was successful.

Original: The RNN network, for example,sometimes (depending on the randomly initialized parameters) was unable to ﬁt the data in a meaningful way (see Figure 9b).
Revised: The RNN network, for example, was unable to ﬁt the data in a meaningful way on some runs (Appendix Figure 14). Figure 10 shows that G(s) for DYRNN5 has more poles.

Original: For each of our experiments, we used a similar count ofparameters in the recurrent part of the network and the same layer constellations (Table 1,5 and 6). Due to the design of our network, its FC part contains more parameters than the other networks.
Revised: This is to be expected because of the structure of the network, but can be an indicator for instability in some frequency areas. G(s) of DYRNN3 follows the given plots better. The Frequency responses of both DYRNN networks are similar to the actual system, but the root locus curve of

Original: Since we omitted an activation function in the FC layer, it amounts to a linear combination ofall channels from the cascaded part.
Revised: DYRNN5 differs signiﬁcantly from the actual one: its starting point is far outside of the plotted region.

Original: Therefore, we assume that the improved performance of our networks stems from the DYRNN computation functions instead of the increased count of parameters in the FC layer. Given a different network conﬁguration and more parameters, it may be possible for the other layer types to achieve results closer to ours for the shown datasets
Revised: On the Heating dataset, the GRU network performed slightly better than DYRNN3 and DYRNN5. It is noticeable that, while the datasets is visually similar to a PT1 element, the DYRNN3 model displays a similar frequency response plot and root locus curve in Figure 11 b) and c) as seen in the actual model in the PT1 system in the Circuit dataset (Figure 10). On the Thermic dataset, DYRNN3 performed similarly to GRU.

Original: [Table]
Revised: The dataset consists of two inputs and one output, yielding two transfer functions G 1 ( s ) and G 2 ( s ) .

Original: [Table]
Revised: These can be analysed separately as shown in Figure 12 for DYRNN3.

Original: [Table]
Revised: While G ( s ) over s is similar to the one in Circuit, the frequency response and root locus of both G 1 ( s ) and G 2 ( s ) are very different from Circuit and Heating. In total, DYRNN5 performed worse than DYRNN3 on the non-simulated datasets. We assume that the reason for this is that Heating and Thermic do not have a strict zero-level at the start and the end of the datasets’ input and output.

Original: [Figure] Our experiments concerning resampling to different sampling rates  show that training for 0.7 ∆ t and predicting to 1.0 ∆ t performs worse than the other two variants.
Revised: That might enforce a more stable model. Our results concerning resampling to different sampling rates (in 9) show that training for 0.7 ∆ t and predicting to 1.0 ∆ t performs worse than the other two variants.

Original: From this we conclude that the total amount of time covered by the training data is more important than the training at different sampling rates for this dataset.
Revised: From this, it can be concluded that the total amount of time covered by the training data is more important than the training at different sampling rates for this dataset.

Original: It also shows that the model is able to extrapolate for each of the different sampling rates to another.
Revised: It also shows that the model is able to extrapolate for each of the different sampling rates to another.

Original: Our implementation which allows the per-batch sampling rate 
Revised: Our implementation, which allows the per-batch sampling rate during training is computationally more expensive than the one which relies upon a ﬁxed sampling rate, so the inexpensive version is a possible alternative for larger datasets.

Original: R
Revised: [Figure]

Original: NN
Revised: NN

Original: L S T
Revised: [Figure]

Original: M
Revised: M

Original: GR U O u r s
Revised: G

Original: [Figure]
Revised: [Figure]

Original: T e s t s e t
Revised: U

Original: M S [Figure]
Revised: DY

Original: R
Revised: R

Original: NN
Revised: NN

Original: L S T M
Revised: [Figure]

Original: G R U O u r s 10 −
Revised: DY

Original: [Figure]
Revised: [Figure]

Original: R

Original: NN
Revised: NN

Original: L S T
Revised: [Figure] [Table] [Figure] [Table]

Original: M
Revised: [Figure] [Table] [Figure]

Original: G
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: u r s [Figure] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] during training is computationally more expensive than the one which relies upon a ﬁxed sampling rate, so the inexpensive version is a possible alternative for larger datasets. [Figure] [Table]
Revised: T e s t s e t

Original: [Figure] [Table] [Figure] [Table]
Revised: M S E

Original: [Table]
Revised: [Figure]

Original:  F UTURE W ORK
Revised:  F UTURE W ORK

Original: We have shown a new type of recurrent layer designed for the modelling of dynamic systems.
Revised: We have shown a new type of recurrent layer designed for the modelling of dynamic systems.

Original: Based on this, we see several potential areas for future work, for example in the domain of automated architecture search.
Revised: Based on this, we see several potential areas for future work.

Original: To enhance the models expressiveness for the user andquantify model uncertainty, a combination with Bayesian methods could also be evaluated. We showed how our network could be interpreted; but have not actually evaluated this yet.
Revised: One point that was not part of this work was the in depth analysis of the learnt transfer functions, and how the transfer function is to be interpreted in case of non-linear activation functions in the network. The plots shown of the transfer functions are accessible and interpretable mainly for engineers.

Original: Here, a study with a variing array of users would be of interest.
Revised: Another area of research would be on how to make these results interpretable for scientists without an engineering background.

Original: Our experiment showed good results on three datasets of system identiﬁcation.
Revised: Our experiment showed competitive results on three datasets of system identiﬁcation.

Original: A different area of application can be model based reinforcement learning tasks, since the layers’ computation blocks are  commonly used in control engineering as well. This should also be evaluated.
Revised: A different area of application can be model based reinforcement learning tasks, since the layers’ computation blocks are also commonly used in control engineering.

Original:  C ONCLUDING R EMARKS
Revised:  C ONCLUDING R EMARKS

Original: In this work we present a new recurrent network architecture which is based on dynamic systems theory.
Revised: In this work we present a new recurrent network architecture which is based on dynamic systems theory.

Original: We show that the learned system dynamics can produce models which can extrapolate to different sampling rates in the data. 
Revised: We show that the learnt system dynamics can produce models which can extrapolate to different sampling rates in the data. Due to the speciﬁc meaning of the cells’ computation function, the [Figure]

Original: Due to the speciﬁc meaning of the cells’ computation function, the learned model can be leveraged to gain insight on the underlying system dynamics.
Revised: [Figure] [Figure] learnt model can be leveraged to gain insight on the underlying system dynamics and a differential equation model of the system can be extracted.

Original: A A PPENDIX B ASE F UNCTIONS IN D YNAMICAL S YSTEMS T HEORY
Revised: A A PPENDIX B ASE F UNCTIONS IN D YNAMICAL S YSTEMS T HEORY

Original: In dynamic systems theory, the basic elements are visualized using their response towards an input unit step function σ ( t ) starting at t=0 with the amplitude of 1.
Revised: In dynamic systems theory, the basic elements are visualized using their response towards an input unit step function σ ( t ) starting at t=0 with the amplitude of 1.

Original: In Table 4, we show a short summary over the blocks used in our network layer.
Revised: In Table 3, we show a short summary over the blocks used in our network layer.

Original: Blocks display a speciﬁc behaviour, which is visualized by looking at the response towards the Unit step function.
Revised: Blocks display a speciﬁc behaviour, which is visualized by looking at the response towards the Unit step function.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: Sequential models keep a hidden state while iterating over an input sequence, to encode information necessary for the computation of the next step’s result.
Revised: Sequential models keep a hidden state while iterating over an input sequence, to encode information necessary for the computation of the next step’s result.

Revised: There are two ways to implement the model: one where the sampling rate is kept constant between batches and one where the sampling rate can be set per batch.

Original: To deal with multiple input channels and the possibility of several base components of the same type, we deﬁne the computations as follows:
Revised: While the ﬁrst version can be implemented using trivial matrix multiplication, the per-batch version is implemented as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: With x ( t ) ’s dimensionality of [batch × sample × input channels] and ∆ t ( t ) as [batch × sample × 1] or ∆ t as [batch × 1 × 1] , the multiplications expressed in Einstein’s sum formation (which can be implemented as described in e.g. https://www.tensorﬂow.org/api docs/python/tf/einsum ) are: ◦ as ’bi, ij → bij’, • as ’bi, b → bi’ and ∗ as ’bij, bij → bij’, i.e. the Hadamard product operator.
Revised: With u ( t ) ’s dimensionality of [batch × sample × input channels] and ∆ t ( t ) as [batch × sample × 1] or ∆ t as [batch × 1 × 1] , the multiplications expressed in Einstein’s sum formation (which can be implemented as described in e.g. https://www.tensorﬂow.org/api docs/python/tf/einsum ) are: ◦ as ’bi, ij → bij’, • as ’bi, b → bi’ and ∗ as ’bij, bij → bij’, i.e. the Hadamard product operator.

Original: The dimensions of the trainable parameters result to
Revised: The dimensions of the trainable parameters result to

Original: [Equation]
Revised: [Equation]

Original: given this setup.
Revised: given this setup.

Original: The weights  were initialized in a random uniform fashion, with a range [0 . 1 , 0 . 2] .
Revised: The weights of P,D,PT1 and PD were initialized in a random uniform distribution, with a range [0 . 1 , 0 . 2] .

Revised: Its important to initialize I such that the ﬁrst iteration does not become unstable.

Revised: For datasets with around 500 elements, the initialization used in the experiments was in the range of [110 .

Revised: 0∆ t, 110 .

Revised: 0∆ t ] .

Original: A. F URTHER E XPERIMENT D OCUMENTATION
Revised: A. F URTHER E XPERIMENT D OCUMENTATION

Original: We evaluated our network on three different datasets.
Revised: We evaluated our network on three different datasets.

Original: One is our own dataset from a Matlab (MATLAB (2018)) simulation as shown in Figure 12, the other two are the ”Heat ﬂow density through a two layer wall” (De Moor B.L.R. (a)) and the ”Heating system” (De Moor B.L.R. (b)) datasets from the DaISy system identiﬁcation benchmarks:
Revised: One is our own dataset from a Matlab (MATLAB (2018)) simulation as shown in Figure 13.

Original: • Heating System Benchmark: Prediction of halogen lamp temperature based on input volt-
Revised: The input of our model is the input voltage of the

Original: age
Revised: System

Original: [Table]
Revised: [Table]

Original: t
Revised: P

Original: [Equation] 
Revised: [Equation] [Equation]

Original: m
Revised: ud

Original: [Equation]
Revised: [Equation]

Original: lit ud e y ( t )
Revised: Y ( s )

Original: [Equation]
Revised: [Equation]

Original: K · u ( t ) (9)

Original: I    t
Revised: I    t

Original: A m p lit ud e˙ y ( t )
Revised: A m p lit ud e

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( t ) (10)
Revised: D

Original: [Table]
Revised: [Equation]

Original: t A m p lit ud ey ( t
Revised: m p lit ud e

Original: [Equation]
Revised: [Equation]

Original: ( t ) (11)

Original: [Table]
Revised: [Equation]

Original: t
Revised: PT

Original: A m
Revised: [Equation] [Equation] m

Original: [Equation]
Revised: [Equation]

Original: lit ude
Revised: ud

Original: T
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: PDt
Revised: PD

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: • Two Walls Benchmark: Prediction of heat ﬂow through two walls based on measurements before and between the walls
Revised: circuit, while the output value is the voltage measured over the capacitor.

Original: For our circuit dataset we trained until overﬁtting on a validation set or until 5000 epochs.
Revised: The simulation’s sampling rate is 0.005s. We simulated three time series for training, validation and testing with length ofseconds. The input consists of a square signal with randomly changing amplitudes from 0.5s to 1.5s. The remainder of the time, the input voltage is 0V.

Original: For the remaining two DaISy datasets, the models were trained for 3000 epochs on a training set without early stopping, because we noticed that for example RNN the training would terminate prematurely, yielding worse results than when trained until completion.
Revised: The other two are the ”Heat ﬂow density through a two layer wall” (De Moor B.L.R. (a)) (in short Thermic) and the ”Heating system” (De Moor B.L.R. (b)) (in short Heating) datasets from the DaISy system identiﬁcation benchmarks: • Heating System Benchmark: Prediction of halogen lamp temperature based on input voltage

Original: After training, the models were evaluated on a testing set split from the complete dataset.
Revised: After training, the models were evaluated on a testing set split from the complete dataset.

Original: The DaISy Datasets to not have a zero-level starting interval like in our circuit dataset.
Revised: The DaISy Datasets do not have a zero-level starting interval like in our circuit dataset.

Original: For simplicity’s sake, our chosen network architecture does not take the initial value of the output time series into account.
Revised: For simplicity’s sake, our chosen network architecture does not take the initial value of the output time series into account.

Original: Therefore an offset could be seen in predictions on the testing set for all layer types.
Revised: Therefore an offset could be seen in predictions on the testing set for all layer types.

Original: In a post-processing step this offset was computed using the median difference between prediction and label data, and subtracted from the prediction prior to the ﬁnal MSE evaluation.
Revised: In a post processing step this offset was computed using the median difference between prediction and label data, and subtracted from the prediction prior to the ﬁnal MSE evaluation.

Original: The amount of units and parameters  are shown in  Tables 5 (Heating
Revised: The amount of units and parameters for all experiments are shown in the Tables 4, 5, 6. [Table] GRU DYRNN

Revised: DYRNN

Original: System) and 6 (Thermic Wall Res.).
Revised: [Table] [Table] per s.c. (7) [Table]

Revised: per s.c. (5)

Revised: [Table]

Revised: [Table]

Revised: per s.

Revised: [Table]

Revised: [Table]

Revised: per

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: GRU

Revised: DYRNN

Revised: DYRNN

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: per s.u. (14)

Revised: [Table]

Revised: s.u. (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: s.u. (

Revised: [Table]

Revised: [Table]

Revised: .u.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: DYRNN

Revised: DYRNN

Revised: [Table]

Revised: [Table]

Revised: per s.u. (14)

Original: [Table]
Revised: [Table]

Revised: per s.u.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: per

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: per

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: A. B EST AND W ORST R ESULTS ON D ATASETS

Revised: The Figures 14, 15 and 16 show the best and worst testing results of all 10 runs per network type.

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: Label

Revised: [Figure]

Revised: .time s

Revised: O u t pu t

Revised: H ea t

Revised: Input

Revised: RNN

Revised: LSTM

Revised: [Figure]

Revised: [Equation]

Revised: Label

Revised: [Figure]

Revised: time s

Revised: [Figure]

Revised: [Figure]

Revised: A. R ESULTING T RANSFER F UNCTIONS

Revised: Below the transfer functions of the best runs are documented.

Revised: The transformation to differential equations is trivial.

Revised: Notice that due to the rounding to 2 signiﬁcant digits, the differential equations listed here are unlikely to yield accurate results.

Revised: Additionally, simpliﬁcation of these transfer functions is possible if the poles and zeros of the function match, but this is outside of the scope of this work.

Revised: The transfer functions of the best runs of DYRNN3 are as follows:

Revised: Circuit:

Revised: [Equation]

Revised: Heating:

Revised: [Equation]

Revised: Thermic Resistance:

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: This sum of transfer functions can be used to simulate y ( t ) by ﬁrst simulating y 1 ( t ) and y 2 ( t ) separatly and then building the sum of the results.