Original: L EARNING TEMPORAL EVOLUTION OF PROBABILITY DISTRIBUTION WITH R ECURRENT N EURAL N ETWORK
Revised: L EARNING TEMPORAL EVOLUTION OF PROBABILITY DISTRIBUTION WITH R ECURRENT N EURAL N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose to tackle a time series regression problem by computing temporal evolution of a probability density function to provide a probabilisitic forecast.
Revised: We propose to tackle a time series regression problem by computing temporal evolution of a probability density function to provide a probabilistic forecast.

Original: A
Revised: A

Original: Recurrent Neural Network (RNN) based model is employed to learn a nonlinear operator for temporal evolution of a probability density function.
Revised: Recurrent Neural Network (RNN) based model is employed to learn a nonlinear operator for temporal evolution of a probability density function.

Original: We use a softmax layer for a numerical discretization of a smooth probability density functions, which transforms a function approximation problem to a classiﬁcation task.
Revised: We use a softmax layer for a numerical discretization of a smooth probability density functions, which transforms a function approximation problem to a classiﬁcation task.

Original: Explicit and implicit regularization strategies are introduced to impose a smoothness condition on the estimated probability distribution.
Revised: Explicit and implicit regularization strategies are introduced to impose a smoothness condition on the estimated probability distribution.

Original: A Monte Carlo procedure to compute the temporal evolution of the distribution for a multiple-step forecast is presented.
Revised: A Monte Carlo procedure to compute the temporal evolution of the distribution for a multiple-step forecast is presented.

Original: The evaluation of the proposed algorithm on two synthetic and two real data sets shows advantage over the compared baselines.
Revised: The evaluation of the proposed algorithm on three synthetic and two real data sets shows advantage over the compared baselines

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Application of the deep learning for manufacturing processes has attracted a great attention as one of the core technologies in Industry 4.0 (Lasi et al., 2014).
Revised: Application of the deep learning for manufacturing processes has attracted a great attention as one of the core technologies in Industry 4.0 (Lasi et al., 2014).

Original: In many manufacturing processes, e.g. blast furnace, smelter, and milling, the complexity of the overall system makes it almost impossible or impractical to develop a simulation model from the ﬁrst principles.
Revised: In many manufacturing processes, e.g. blast furnace, smelter, and milling, the complexity of the overall system makes it almost impossible or impractical to develop a simulation model from the ﬁrst principles.

Original: Hence, system identiﬁcation from sensor observations has been a long-standing research topic (Wang et al., 2016).
Revised: Hence, system identiﬁcation from sensor observations has been a long-standing research topic (Wang et al., 2016).

Original: Still, when the observation is noisy and there is no prior knowledge on the underlying dynamics, there is only a very limited number of methods for the reconstruction of nonlinear dynamics.
Revised: Still, when the observation is noisy and there is no prior knowledge on the underlying dynamics, there is only a very limited number of methods for the reconstruction of nonlinear dynamics.

Original: In this work, we consider the following class of problems, where the system is driven by a complex underlying dynamical system, e.g.,
Revised: In this work, we consider the following class of problems, where the system is driven by a complex underlying dynamical system, e.g.,

Original: [Equation]
Revised: [Equation]

Original: Here, y ( t ) is a continuous process, F is a nonlinear operator, τ is a delay-time parameter, and u ( t ) is an exogenous forcing, such as control parameters.
Revised: Here, y ( t ) is a continuous process, F is a nonlinear operator, τ is a delay-time parameter, and u ( t ) is an exogenous forcing, such as control parameters.

Original: At time step t , we then observe a noisy measurement of y ( t ) which can be deﬁned by the following noise model
Revised: At time step t , we then observe a noisy measurement of y ( t ) which can be deﬁned by the following noise model

Original: [Equation]
Revised: [Equation]

Original: where ν t is a multiplicative and (cid:15) t is an additive noise process.
Revised: where ν t is a multiplicative and (cid:15) t is an additive noise process.

Original: In (1) and (2), we place no assumption on function F , do not assume any distributional properties of noises ν t and (cid:15) t , but assume the knowledge of the control parameters u ( t ) .
Revised: In (1) and (2), we place no assumption on function F , do not assume any distributional properties of noises ν t and (cid:15) t , but assume the knowledge of the control parameters u ( t ) .

Original: Since the noise components, ν t and (cid:15) t , are stochastic processes, the observation ˆ y t is a random variable.
Revised: Since the noise components, ν t and (cid:15) t , are stochastic processes, the observation ˆ y t is a random variable.

Original: In this work, we are interested in computing temporal evolution of the probability density function (PDF) of ˆ y , given the observations up to time step t , i.e., p (ˆ y t + n |
Revised: In this work, we are interested in computing temporal evolution of the probability density function (PDF) of ˆ y , given the observations up to time step t , i.e., p (ˆ y t + n |

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: 1 ) for n ≥ 1 , where (cid:98)
Revised: 1 ) for n ≥ 1 , where (cid:98)

Original: Y 0:
Revised: Y 0:

Original: t =
Revised: t =

Original: (ˆ y 0 , · · · , ˆ y t ) is a trajectory of the past observations and U 0:
Revised: (ˆ y 0 , · · · , ˆ y t ) is a trajectory of the past observations and U 0:

Original: t + n − 1 = ( u 0 , · · · , u t + n − 1 ) consists of the history of the known control actions, U 0:
Revised: t + n − 1 = ( u 0 , · · · , u t + n − 1 ) consists of the history of the known control actions, U 0:

Original: t − 1 , and a future control scenario, U t : t + n − 1 .
Revised: t − 1 , and a future control scenario, U t : t + n − 1 .

Original: We show, in Section 3, a class of problems, where simple regression problem of forecasting the value of ˆ y t + n is not sufﬁcient or not possible, e.g., chaotic systems.
Revised: We show, in Section 3, a class of problems, where simple regression problem of forecasting the value of ˆ y t + n is not sufﬁcient or not possible, e.g., chaotic systems.

Original: Note that the computation of time evolution of a PDF has been a long-standing topic in statistical physics.
Revised: Note that the computation of time evolution of a PDF has been a long-standing topic in statistical physics.

Original: For a simple Markov process, there are well-established theories based on the Fokker-Planck equation.
Revised: For a simple Markov process, there are well-established theories based on the Fokker-Planck equation.

Original: However, it is very difﬁcult to extend those theories to a more general problem, such as delay-time dynamical systems, or apply it to complex nonlinear systems.
Revised: However, it is very difﬁcult to extend those theories to a more general problem, such as delay-time dynamical systems, or apply it to complex nonlinear systems.

Original: Modeling of the system (1) has been extensively studied in the past, in particular, under the linearity assumptions on F and certain noise models, e.g., Gaussian (cid:15) t and ν t = 1 in (2).
Revised: Modeling of the system (1) has been extensively studied in the past, in particular, under the linearity assumptions on F and certain noise models, e.g., Gaussian (cid:15) t and ν t = 1 in (2).

Original: The approaches based on auto-regressive processes (L¨utkepohl, 2005) and Kalman ﬁlter (Harvey, 1990) are good examples.
Revised: The approaches based on auto-regressive processes (L¨utkepohl, 2005) and Kalman ﬁlter (Harvey, 1990) are good examples.

Original: Although these methods do estimate the predictive probability distribution and enable the computation of the forecast uncertainty, the assumptions on the noise and linearity in many cases make it challenging to model real nonlinear dynamical systems. 
Revised: Although these methods do estimate the predictive probability distribution and enable the computation of the forecast uncertainty, the assumptions on the noise and linearity in many cases make it challenging to model real nonlinear dynamical systems. Recently, a nonlinear state-space model based on the Gaussian process, called the Gaussian Process

Revised: State Space Model (GPSSM), has been extended for the identiﬁcation of nonlinear system (Frigola et al., 2013; Eleftheriadis et al., 2017).

Revised: GPSSM is capable of representing a nonlinear system and is particularly advantageous when the size of the data set is relatively small that it is difﬁcult to train a deep learning model.

Revised: However, the joint Gaussian assumption of GPSSM may restrict the representation capability for a complex non-Gaussian noise.

Original: A recent success of deep learning created a ﬂurry of new approaches for time series modeling and prediction.
Revised: A recent success of deep learning created a ﬂurry of new approaches for time series modeling and prediction.

Original: The ability of deep neural networks, such as RNN, to learn complex nonlinear spatiotemporal relationships in the data enabled these methods to outperform the classical time series approaches.
Revised: The ability of deep neural networks, such as RNN, to learn complex nonlinear spatiotemporal relationships in the data enabled these methods to outperform the classical time series approaches.

Original: For example, in the recent works of Qin et al.
Revised: For example, in the recent works of Qin et al.

Original: Hsu (2017); Dasgupta & Osogami (2017), the authors proposed different variants of the RNN-based algorithms to perform time series predictions and showed their advantage over the traditional methods.
Revised: Hsu (2017); Dasgupta & Osogami (2017), the authors proposed different variants of the RNN-based algorithms to perform time series predictions and showed their advantage over the traditional methods.

Original: Although encouraging, these approaches lack the ability to estimate the probability distribution of the predictions since RNN is a deterministic model and unable to fully capture the stochastic nature of the data.
Revised: Although encouraging, these approaches lack the ability to estimate the probability distribution of the predictions since RNN is a deterministic model and unable to fully capture the stochastic nature of the data.

Original: To enable RNN to model the stochastic properties of the data, Chung et al.
Revised: To enable RNN to model the stochastic properties of the data, Chung et al.

Original: (2015) augmented RNN with a latent random variable included in the hidden state and proposed to estimate the resulting model using variational inference.
Revised: (2015) augmented RNN with a latent random variable included in the hidden state and proposed to estimate the resulting model using variational inference.

Original: In a similar vein, the works of Archer et al.
Revised: In a similar vein, the works of Archer et al.

Original: ; Krishnan et al.
Revised: ; Krishnan et al.

Original: (2017) extend the traditional Kalman ﬁlter to handle nonlinear dynamics when the inference becomes intractable.
Revised: (2017) extend the traditional Kalman ﬁlter to handle nonlinear dynamics when the inference becomes intractable.

Original: Their approach is based on formulating the variational lower bound and optimizing it under the assumption of Gaussian posterior.
Revised: Their approach is based on formulating the variational lower bound and optimizing it under the assumption of Gaussian posterior.

Original: Another recent line of works enabled stochasticity in the RNN-based models by drawing a connection between Bayesian variation inference and a dropout technique.
Revised: Another recent line of works enabled stochasticity in the RNN-based models by drawing a connection between Bayesian variation inference and a dropout technique.

Original: In particular, Gal & Ghahramani (2016) showed that the model parameter uncertainty (which then leads to uncertainty in model predictions), that traditionally was estimated using variational inference, can be approximated using a dropout method (a random removal of some connections in the network structure).
Revised: In particular, Gal & Ghahramani (2016) showed that the model parameter uncertainty (which then leads to uncertainty in model predictions), that traditionally was estimated using variational inference, can be approximated using a dropout method (a random removal of some connections in the network structure).

Original: The prediction uncertainty is then estimated by evaluating the model outputs at different realizations of the dropout weights.
Revised: The prediction uncertainty is then estimated by evaluating the model outputs at different realizations of the dropout weights.

Original: Following the ideas of Gal & Ghahramani (2016), Zhu & Laptev (2017) proposed additional ways (besides modeling the parameter uncertainty) to quantify the forecast uncertainty in
Revised: Following the ideas of Gal & Ghahramani (2016), Zhu & Laptev (2017) proposed additional ways (besides modeling the parameter uncertainty) to quantify the forecast uncertainty in

Original: RNN, which included the model mis-speciﬁcation error and the inherent noise of the data.
Revised: RNN, which included the model mis-speciﬁcation error and the inherent noise of the data.

Original: 1. O VERVIEW OF THE PROPOSED WORK
Revised: 1. O VERVIEW OF THE PROPOSED WORK

Original: We propose an RNN-model to compute the temporal evolution of a PDF, p (ˆ y t + n | (cid:98)
Revised: We propose an RNN-model to compute the temporal evolution of a PDF, p (ˆ y t + n | (cid:98)

Original: Y 0: t , U 0: t + n − 1 ) .
Revised: Y 0: t , U 0: t + n − 1 ) .

Original: To avoid the difﬁculties in directly estimating the continuous function, we use a numerical discretization technique, which converts the function approximation problem to a classiﬁcation task (see Section 2.2).
Revised: To avoid the difﬁculties in directly estimating the continuous function, we use a numerical discretization technique, which converts the function approximation problem to a classiﬁcation task (see Section 2.2).

Original: We note that the use of the traditional cross-entropy (CE) loss in our formulated classiﬁcation problem can be problematic since it is oblivious to the class ordering.
Revised: We note that the use of the traditional cross-entropy (CE) loss in our formulated classiﬁcation problem can be problematic since it is oblivious to the class ordering.

Original: To address this, we additionally propose two regularizations for CE to account for a geometric proximity between the classes (see Sections 2.2.1 and 2.2.2).
Revised: To address this, we additionally propose two regularizations for CE to account for a geometric proximity between the classes (see Sections 2.2.1 and 2.2.2).

Original: The probability distribution of one-step-ahead prediction, p (ˆ y t +1 | (cid:98)
Revised: The probability distribution of one-step-ahead prediction, p (ˆ y t +1 | (cid:98)

Original: Y 0: t , U 0: t ) can now be simply estimated from the output softmax layer of RNN (see Section 2.2), while to propagate the probability distribution further in time, for a multiple-step forecast, we propose a sequential Monte Carlo (SMC) method (see Section 2.3).
Revised: Y 0: t , U 0: t ) can now be simply estimated from the output softmax layer of RNN (see Section 2.2), while to propagate the probability distribution further in time, for a multiple-step forecast, we propose a sequential Monte Carlo (SMC) method (see Section 2.4). For clarity, we present most derivations for univariate time series but also show the extension to multivariate data in Section 2.3.

Original: We empirically show that the proposed modeling approach enables us to represent a continuous PDF of any arbitrary shape, including the ability to handle the multiplicative data noises in (2).
Revised: We empirically show that the proposed modeling approach enables us to represent a continuous PDF of any arbitrary shape, including the ability to handle the multiplicative data noises in (2).

Original: Since the probability distribution is computed, the RNN-model can also be used for a regression task by computing the expectation (see Section 2.3).
Revised: Since the probability distribution is computed, the RNN-model can also be used for a regression task by computing the expectation (see Section 2.4).

Original: Hereafter, we use DE-RNN for the proposed RNN model, considering the similarity with the density-estimation task.
Revised: Hereafter, we use DE-RNN for the proposed RNN model, considering the similarity with the density-estimation task.

Original: In summary, the contributions of this work are as follows: (i) formulate the classical regression problem for time series prediction as a predictive density-estimation problem, which can be solved by a classiﬁcation task (ii) propose an approach to compute the time evolution of probability distribution using SMC on the distributions from DE-RNN (iii) proposed two regularizations for CE loss to capture the ordering of the classes in the discretized PDF.
Revised: In summary, the contributions of this work are as follows: (i) formulate the classical regression problem for time series prediction as a predictive density-estimation problem, which can be solved by a classiﬁcation task (ii) propose an approach to compute the time evolution of probability distribution using SMC on the distributions from DE-RNN (iii) proposed two regularizations for CE loss to capture the ordering of the classes in the discretized PDF.

Original: We evaluated the proposed algorithm on two synthetic and two real datasets, showing its advantage over the baselines.
Revised: We evaluated the proposed algorithm on three synthetic and two real datasets, showing its advantage over the baselines.

Original: Note that DE-RNN has a direct relevance to a wide range of problems in physics and engineering, in particular, for uncertainty quantiﬁcation and propagation (Zhang & Karniadakis, 2017).
Revised: Note that DERNN has a direct relevance to a wide range of problems in physics and engineering, in particular, for uncertainty quantiﬁcation and propagation (Zhang & Karniadakis, 2017).

Original:  LSTM FOR N OISY D YNAMICAL S YSTEM
Revised:  LSTM FOR N OISY D YNAMICAL S YSTEM

Original: In this Section we present the details of the proposed algorithm using a speciﬁc form of RNN, called Long Short-Term Memory (LSTM) network.
Revised: In this Section we present the details of the proposed algorithm using a speciﬁc form of RNN, called Long Short-Term Memory (LSTM) network.

Original: Although in the following presentation and experiments we used LSTM, other networks, e.g., GRU (Chung et al., 2014), can be used instead.
Revised: Although in the following presentation and experiments we used LSTM, other networks, e.g., GRU (Chung et al., 2014), can be used instead.

Original: 2. R EVIEW OF L ONG S HORT -T ERM M EMORY NETWORK
Revised: 2. R EVIEW OF L ONG S HORT -T ERM M EMORY NETWORK

Original: The Long Short-Term Memory network (LSTM) (Hochreiter & Schmidhuber, 1997; Gers et al., 2000) consists of a set of nonlinear transformations of input variables z t ∈ R m ; 
Revised: The Long Short-Term Memory network (LSTM) (Hochreiter & Schmidhuber, 1997; Gers et al., 2000) consists of a set of nonlinear transformations of input variables z t [Equation] R m ; Gating functions

Original: [Equation]
Revised: [Equation]

Original: Here, ϕ S and ϕ T , respectively, denote the sigmoid and hyperbolic tangent functions, L is a linear layer, which includes a bias, s t ∈ R N c is the internal state, h t ∈ R N c is the output of the LSTM network, N c is the number of the LSTM units, and a (cid:12) b denote a component-wise multiplication.
Revised: Here, ϕ S and ϕ T , respectively, denote the sigmoid and hyperbolic tangent functions, L is a linear layer, which includes a bias, s t ∈ R N c is the internal state, h t ∈ R N c is the output of the LSTM network, N c is the number of the LSTM units, and a (cid:12) b denote a component-wise multiplication.

Original: Interesting observation can be made about equation (4).
Revised: Interesting observation can be made about equation (4).

Original: We can re-write equation (4) as
Revised: We can re-write equation (4) as

Original: [Equation]
Revised: [Equation]

Original:  for some functions f and g .
Revised: 6) for some functions f and g .

Original: With a simple re-scaling, this equation can be interpreted as a ﬁrst-order
Revised: With a simple re-scaling, this equation can be interpreted as a ﬁrst-order

Original: Euler scheme for a linear dynamical system,
Revised: Euler scheme for a linear dynamical system,

Original: [Equation]
Revised: [Equation]

Original: Thus, LSTM can be understood as a series expansion, where a complex nonlinear dynamical system is approximated by a combination of many simpler dynamical systems.
Revised: Thus, LSTM can be understood as a series expansion, where a complex nonlinear dynamical system is approximated by a combination of many simpler dynamical systems.

Original: Usually, LSTM network is supplemented by feed-forward neural networks, e.g.,
Revised: Usually, LSTM network is supplemented by feed-forward neural networks, e.g.,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: in which x t is the input feature.
Revised: in which x t is the input feature.

Original: Using (5), we can denote by Ψ e and Ψ d a collection of the operators from input to internal state (encoder) and from internal state to the output o (decoder):
Revised: Using (5), we can denote by Ψ e and Ψ d a collection of the operators from input to internal state (encoder) and from internal state to the output P (decoder):

Original: [Equation]
Revised: [Equation]

Original: 2. D ISCRETE A PPROXIMATION OF P ROBABILITY D ENSITY F UNCTION
Revised: 2. D ISCRETE A PPROXIMATION OF P ROBABILITY D ENSITY F UNCTION

Original: In this Section we ﬁrst consider the problem of modeling the conditional PDF, p (ˆ y t +1 |(cid:98) Y
Revised: In this Section we ﬁrst consider the problem of modeling the conditional PDF, p (ˆ y t +1 |

Original: [Equation]
Revised: [Equation]

Original: t
Revised: Y

Original: [Equation]
Revised: [Equation]

Original:  U 0: [Equation]
Revised: t , U 0: t ) .

Original: Although ˆ y t +1 has a dependence on the past trajectories of both ˆ y and u , using the “state space”
Revised: Although ˆ y t +1 has a dependence on the past trajectories of both ˆ y and u , using the “state space”

Original: LSTM model argument in Section 2.1, the conditional PDF can be modeled as a Markov process
Revised: LSTM model argument in Section 2.1, the conditional PDF can be modeled as a Markov process

Original: [Equation]
Revised: [Equation]

Original: Hence, to simplify the problem, we consider a task of estimating the PDF of a random variable ˆ y , given an input x , i.e., p (ˆ y | x ) .
Revised: Hence, to simplify the problem, we consider a task of estimating the PDF of a random variable ˆ y , given an input x , i.e., p (ˆ y | x ) .

Original: The obtained results can then be directly applied to the original problem of estimating p (ˆ y t +1 | s t ) .
Revised: The obtained results can then be directly applied to the original problem of estimating p (ˆ y t +1 | s t ) .

Original: Let α = ( α 0 , · · · , α K ) denote a set of real numbers, such that α i − 1 <
Revised: Let α = ( α 0 , · · · , α K ) denote a set of real numbers, such that α i − 1 <

Original: α i for i = 1 , · · · , K , which deﬁnes K disjoint intervals,
Revised: α i for i = 1 , · · · , K , which deﬁnes K disjoint intervals,

Original: Ii
Revised: I

Original: [Equation]
Revised: [Equation]

Original: i − 1 , α i ) .
Revised: i [Equation] 1 , α i ) .

Original: Then, a discrete probability distribution can be deﬁned
Revised: Then, a discrete probability distribution can be deﬁned

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where it is clear that p ( k | x ) is a numerical discretization of the continuous PDF, p (ˆ y | x ) .
Revised: where it is clear that p ( k | x ) is a numerical discretization of the continuous PDF, p (ˆ y | x ) .

Original: Using the
Revised: Using the

Original: LSTM from Section 2.1, the discrete probability p ( k | x ) can be modeled by the softmax layer ( P ) as an output of Ψ d in (9) such that
Revised: LSTM from Section 2.1, the discrete probability p ( k | x ) can be modeled by the softmax layer ( P ) as an output of Ψ d in (9) such that

Original: [Equation]
Revised: [Equation]

Original: Thus, the original problem of estimating a smooth function, p (ˆ y | x ) , is transformed into a classiﬁcation problem of estimating p ( k | x ) in a discrete space.
Revised: Thus, the original problem of estimating a smooth function, p (ˆ y | x ) , is transformed into a classiﬁcation problem of estimating p ( k | x ) in a discrete space.

Original: Obviously, the size of the bin, |I
Revised: Obviously, the size of the bin, |I

Original: j | , affects the ﬁdelity of the approximation.
Revised: j | , affects the ﬁdelity of the approximation.

Original: The effects of the bin size are presented in Section 3.1.
Revised: The effects of the bin size are presented in Section 3.1.

Original: There is a similarity between the discretization and the idea of Lin et al.
Revised: There is a similarity between the discretization and the idea of Lin et al.

Original: However, it should be noted that the same discretization technique, often called “ﬁnite volume method”, has been widely used in the numerical simulations of partial differential equations for a long time.
Revised: However, it should be noted that the same discretization technique, often called “ﬁnite volume method”, has been widely used in the numerical simulations of partial differential equations for a long time.

Original: The discretization naturally leads to the conventional cross-entropy (CE) minimization.
Revised: The discretization naturally leads to the conventional cross-entropy (CE) minimization.

Original: Suppose we have a data set, D R = { (ˆ y i , x i );
Revised: Suppose we have a data set, D R = { (ˆ y i , x i );

Original: ˆ y i ∈ R , x i ∈ R , and i = 1 , . . .
Revised: ˆ y i ∈ R , x i ∈ R , and i = 1 , . . .

Original: , N } .
Revised: , N } .

Original: We can deﬁne a mapping
Revised: We can deﬁne a mapping

Original: C : R → N + such that C (ˆ y )
Revised: C : R → N + such that C (ˆ y )

Original: =
Revised: =

Original: k, if y ∈ I k .
Revised: k, if y ∈ I k .

Original: Then, D R can be easily converted to a new data set for target labels, D
Revised: Then, D R can be easily converted to a new data set for target labels, D

Original: C = { ( c i , ˆ y i , x i ); c i ∈ N + , ˆ
Revised: C = { ( c i , ˆ y i , x i ); c i ∈ N + , ˆ

Original: y i ∈
Revised: y i ∈

Original: R , x i ∈ R , and i = 1 , . . .
Revised: R , x i ∈ R , and i = 1 , . . .

Original: , N } , where c i =
Revised: , N } , where c i =

Original: C
Revised: C

Original: (ˆ y i ) .
Revised: (ˆ y i ) .

Original: D
Revised: D

Original: C provides a training data set for the following CE minimization problem,
Revised: C provides a training data set for the following CE minimization problem,

Original: [Equation]
Revised: [Equation]

Original: Note, however, that the CE minimization does not explicitly guarantee the smoothness of the estimated distribution.
Revised: Note, however, that the CE minimization does not explicitly guarantee the smoothness of the estimated distribution.

Original: Since CE loss function depends only on P i of a correct label, δ c n k , as a result, in the optimization problem every element P i , except for the one corresponding to the correct label,
Revised: Since CE loss function depends only on P i of a correct label, δ c n k , as a result, in the optimization problem every element P i , except for the one corresponding to the correct label,

Original: P c n , is penalized in the same way, which is natural in the conventional classiﬁcation tasks where a geometric proximity between the classes is not relevant.
Revised: P c n , is penalized in the same way, which is natural in the conventional classiﬁcation tasks where a geometric proximity between the classes is not relevant.

Original: In the present study, however, the softmax layer, or class probability, is used as a discrete approximation to a smooth function.
Revised: In the present study, however, the softmax layer, or class probability, is used as a discrete approximation to a smooth function.

Original: Hence, it is expected that P c n and P c n ± 1 (i.e., the nearby classes) should be close to each other.
Revised: Hence, it is expected that P c n and P c n ± 1 (i.e., the nearby classes) should be close to each other.

Original: To address this issue, in the following Sections 2.2.1 and 2.2.2, we propose two types of regularization to impose the class proximity structure in the CE loss.
Revised: To address this issue, in the following Sections 2.2.1 and 2.2.2, we propose two types of regularization to impose the class proximity structure in the CE loss.

Original: 2.2. E XPLICIT R EGULARIZATION OF C ROSS -E NTROPY L OSS
Revised: 2.2. E XPLICIT R EGULARIZATION OF C ROSS -E NTROPY L OSS

Original: To explicitly impose the smoothness between the classes, we propose to use a regularized crossentropy (RCE) minimization, deﬁned by the following loss function 
Revised: To explicitly impose the smoothness between the classes, we propose to use a regularized crossentropy (RCE) minimization, deﬁned by the following loss function [Equation]

Original: [Equation]
Revised: [Equation]

Original: where λ is a penalty parameter and the Laplacian matrix L ∈ R K − 2 ,K is
Revised: where λ is a penalty parameter and the Laplacian matrix L ∈ R K − 2 ,K is

Original: [Equation]
Revised: [Equation]

Original: RCE is analogous to the penalized maximum likelihood solution for density estimation (Silverman, 1986).
Revised: RCE is analogous to the penalized maximum likelihood solution for density estimation (Silverman, 1986).

Original: Assuming a uniform bin size, |I 

Revised: Assuming a uniform bin size, |I

Original: | = · · · = |I K
Revised: | = · · · = |I K

Original: | =
Revised: | =

Original: δy , the Laplacian of a distribution can be approximated by a Taylor expansion p (cid:48)(cid:48)
Revised: δy , the Laplacian of a distribution can be approximated by a Taylor expansion p (cid:48)(cid:48)

Original: (ˆ y| x
Revised: (ˆ y

Original: [Equation]
Revised: [Equation]

Original: |
Revised: y =

Original: [Equation] = [Equation]
Revised: α

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  2 (cid:39) ( P i − 1 − 2 P
Revised: 1 / 2 (cid:39) ( P i − 1 − 2 P

Original: i
Revised: i

Original: + P
Revised: + P

Original: i +1 ) /δy
Revised: i +1 ) /δy

Original: 2 , where α i − 1 / 2 = 0 .
Revised: 2 , where α i − 1 / 2 = 0 .

Original: 5( α i − 1 + α i ) .
Revised: 5( α i − 1 + α i ) .

Original: Then, it is clear that
Revised: Then, it is clear that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: In other words, RCE aims to smooth out the distribution by penalizing local minima or maxima.
Revised: In other words, RCE aims to smooth out the distribution by penalizing local minima or maxima.

Original: 2.2. I MPLICIT R EGULARIZATION OF C ROSS -E NTROPY L OSS
Revised: 2.2. I MPLICIT R EGULARIZATION OF C ROSS -E NTROPY L OSS

Original: Alternative to adding an explicit regularization to CE, the smoothness can be achieved by enforcing a spatial correlation in the network output.
Revised: Alternative to adding an explicit regularization to CE, the smoothness can be achieved by enforcing a spatial correlation in the network output.

Original: Here, we use an one-dimensional convolution layer to enforce smoothness.
Revised: Here, we use an one-dimensional convolution layer to enforce smoothness.

Original: Let (cid:101) o ∈ R K be the input to the softmax layer in DE-DNN.
Revised: Let (cid:101) o ∈ R K denote the last layer of DE-RNN, which was the input to the softmax layer.

Original: We can add a convolution layer, o ∈ R K , on top of
Revised: We can add a convolution layer, o ∈ R K , on top of

Original: [Equation]
Revised: [Equation]

Original: o , such that
Revised: o , such that

Original: [Equation]
Revised: [Equation]

Original: where thepenalty parameter h determines the smoothness of the estimated distribution. 
Revised: where the parameter h determines the smoothness of the estimated distribution. Then, o is supplied to the softmax layer.

Original: Using (17), the LSTM model can now be trained by the standard CE.
Revised: Using (17), DE-RNN can now be trained by the standard CE.

Original: The implicit regularization, here we call convolution CE (CCE), is analogous to a kernel density estimation. 
Revised: The implicit regularization, here we call convolution CE (CCE), is analogous to a kernel density estimation. 2. M ULTIVARIATE T IME S ERIES

Revised: In the modeling of multivariate time series, it is usually assumed that the noise is independent, i.e., the covariance of the noise is a diagonal matrix.

Original: 2. C OMPUTING T IME E VOLUTION OF P ROBABILITY D ISTRIBUTION
Revised: In this case, it is straightforward to extend DERNN, by using multiple softmax layers as the output of DE-RNN. However, such an independent noise assumption signiﬁcantly limits the representative capability of an RNN. Here, we propose to use a set of independently trained DE-RNNs to compute the joint PDF of a multivariate time series. Let ˆ y t be a l -dimensional multivariate time series;

Revised: ˆ y t =

Revised: (ˆ y (1) t , · · · , ˆ y ( l ) t ) .

Revised: The joint PDF can be represented by a product rule,

Revised: [Equation]

Revised: where the dependency on the past trajectory ( (cid:98)

Revised: Y 0: t , U 0: t ) is omitted in the notation for simplicity.

Revised: Directly learning the joint PDF, p ( ˆ y t +1 | (cid:98)

Revised: Y 0: t , U 0: t ) , in a tensor product space is not scalable.

Revised: Instead, a set of DE-RNN is trained to represent the conditional PDFs shown on the right hand side of the above expression.

Revised: Then, the joint PDF can be computed by a product of the Softmax outputs of the

Revised: DE-RNNs.

Revised: Note that, although it requires training l DE-RNNs to compute the full joint PDF, there is no dependency between the DE-RNNs in the training phase.

Revised: So, the set of DE-RNNs can be trained in parallel, which can greatly reduce the training time.

Revised: The details of the multivariate DE-RNN are explained in Appendix A.

Original: In the next-step prediction of DE-RNN, the inputs  are (ˆ y t , u t ) , and the output is the probability distribution,
Revised: The inputs to a DE-RNN are (ˆ y t , u t ) , and the output is the probability distribution,

Original: [Equation]
Revised: [Equation]

Original: Note that D C is used only in the training stage.
Revised: Note that D C is used only in the training stage.

Original: Then, the moments of the predictive distribution can be easily evaluated, e.g.,
Revised: Then, the moments of the predictive distribution can be easily evaluated, e.g.,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where α 1 / 2 = ( α 1 / 2 , α 1+1 / 2 , · · · , α K − 1 / 2 ) T , α 21 / 2 = α 1 / 2 (cid:12) α 1 / 2 ,
Revised: where α 1 / 2 = ( α 1 / 2 , α 1+1 / 2 , · · · , α K − 1 / 2 ) T , α 21 / 2 = α 1 / 2 (cid:12) α 1 / 2 ,

Original: and α i − 1 / 2 = 0 .
Revised: and α i − 1 / 2 = 0 .

Original: 5( α i − 1 + α i ) .
Revised: 5( α i − 1 + α i ) .

Original: Next, we consider a multiple-step forecast, which corresponds to computing a temporal evolution of the probability distribution, i.e., p (ˆ y t + n | 
Revised: Next, we consider a multiple-step forecast, which corresponds to computing a temporal evolution of the probability distribution, i.e., p (ˆ y t + n | (cid:98) Y 0: t , U 0: t + n − 1 ) for n > 1 .

Original: [Equation]
Revised: For simplicity, the multiple-step forecast is shown only for a univariate time series. An extension to a multivariate time series is straightforward (Appendix A).

Original: for n > 1 . Applying the results of Section 2.2, once the distribution of ˆ y t +1 in (10) is computed, the distribution of ˆ y t +2 can be similarly obtained as p (ˆ y t +2 | s t +1 ) .
Revised: Applying the results of Section 2.2, once the distribution of ˆ y t +1 in (10) is computed, the distribution of ˆ y t +2 can be similarly obtained as p (ˆ y t +2 | s t +1 ) .

Original: Observe that s t +1 is computed from a deterministic function of s t , u t +1 , and ˆ y t +1 , i.e.,[-s t +
-]
Revised: Observe that s t +1 is computed from a deterministic function of s t , u t +1 , and ˆ y t +1 , i.e.,[-s t +
-]

Original: =
Revised: [Equation]

Original: Ψ e (ˆ y t +1 , u t +1 , s t ) . Here, u t +1 and s t are already known, while ˆ y t +1 is a random variable, whose distribution p (ˆ y t +1 | s t ) is computed from the deterministic function
Revised: Here, u t +1 and s t are already known, while ˆ y t +1 is a random variable, whose distribution p (ˆ y t +1 | s t ) is computed from the deterministic function

Original: Ψ d ( s t ) .
Revised: Ψ d ( s t ) .

Original: Then, s t +1 is also a random variable.
Revised: Then, s t +1 is also a random variable.

Original: The distribution, p ( s t +1 | s t , u t +1 ) , can be obtained by applying a change of variables on p (ˆ y t +1 | s t ) with a nonlinear mapping Ψ e .
Revised: The distribution, p ( s t +1 | s t , u t +1 ) , can be obtained by applying a change of variables on p (ˆ y t +1 | s t ) with a nonlinear mapping Ψ e .

Original: Repeating this process, the multiple-step-ahead predictive distribution can therefore be computed as
Revised: Repeating this process, the multiple-step-ahead predictive distribution can therefore be computed as

Original: [Equation]
Revised: [Equation]

Original: Since the high dimensional integration in (19) is intractable, we propose to approximate it by a sequential Monte Carlo method.
Revised: Since the high dimensional integration in (19) is intractable, we propose to approximate it by a sequential Monte Carlo method.

Original: The Monte Carlo procedure is outlined in Algorithm 1.Algorithm
Revised: The Monte Carlo procedure is outlined in Algorithm 1.

Original: Input : (cid:98)
Revised: Input : (cid:98)

Original: Y 0: t , U 0: t , number of Monte Carlo samples, N s , and forecast horizon n
Revised: Y 0: t , U 0: t , number of Monte Carlo samples, N s , and forecast horizon n

Original: Output : p (ˆ y t + n | (cid:98)
Revised: Output : p (ˆ y t + n | (cid:98)

Original: Y 0: t , U 0: t + n − 1 ) (density estimation from ˆ y t + n )
Revised: Y 0: t , U 0: t + n − 1 ) (density estimation from ˆ y t + n )

Original: Initialization: Set LSTM states to s 0 = h 0 =
Revised: Initialization: Set LSTM states to s 0 = h 0 =

Original: Perform a sequential update of LSTM up to time t from the noisy observations ( (cid:98)
Revised: Perform a sequential update of LSTM up to time t from the noisy observations ( (cid:98)

Original: Y 0: t ).
Revised: Y 0: t ).

Original: s i =
Revised: s i =

Original: Ψ e (ˆ y i , u i , s i − 1 ) for i = 1 , · · · , t.
Revised: Ψ e (ˆ y i , u i , s i − 1 ) for i = 1 , · · · , t.

Original: MakeN s replicasof the internal state, s 1 t = · · · = sN s t = 
Revised: Make replicas s s t = repeat

Original: s t . repeat Computethe predictive distribution of ˆ y it +1 for each sample P
Revised: Compute

Original: [Equation]
Revised: [Equation]

Original: ) , for i = 1 , · · · , N s .

Original: Sample the target variable at t + 1 , ˆ y

Original: it +1 , from the distribution 1.

Original: Sample the class label from the discrete distribution:

Original: c i ∼ P it + 2. Sample ˆ y i t +1 in I c i : ˆ y i t +1∼ U ( α c i − 1 , α c i ) Update the internal state of LSTM
Revised: , u t +1 , s it ) .

Original: [Equation]

Original: [Equation]

Original: until (all ˆ y t + n are sampled)
Revised: until (all ˆ y t + n are sampled)

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, DE-RNN is tested against two synthetic and two real data sets.
Revised: In this section, DE-RNN is tested against three synthetic and two real data sets.

Original: The LSTM architecture used in all of the numerical experiments is identical.
Revised: The LSTM architecture used in all of the numerical experiments is identical.

Original: Two feed-forward networks are used before and after the LSTM;
Revised: Two feed-forward networks are used before and after the LSTM;

Original: [Equation]
Revised: [Equation]

Original: in which ϕ SP and ϕ SM denote the softplus and softmax functions, respectively.
Revised: in which ϕ SP and ϕ SM denote the softplus and softmax functions, respectively.

Original: The size of the bins is kept uniform, i.e., |I 1 | = · · · = |I K |
Revised: The size of the bins is kept uniform, i.e., |I 1 | = · · · = |I K |

Original: = δy .
Revised: = δy .

Original: The LSTM is trained by using ADAM (Kingma & Ba, 2015) with a minibath size of 20 and a learning rate of η = 10 − 3 .
Revised: The LSTM is trained by using ADAM (Kingma & Ba, 2015) with a minibath size of 20 and a learning rate of η = 10 − 3 .

Original: 3. C OX -I NGERSOLL -R OSS PROCESS
Revised: 3. C OX -I NGERSOLL -R OSS PROCESS

Original: First, we consider a modiﬁed Cox-Ingersoll-Ross (CIR) process, which is represented by the following stochastic differential equation,
Revised: First, we consider a modiﬁed Cox-Ingersoll-Ross (CIR) process, which is represented by the following stochastic differential equation,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: in which W is the Weiner process.
Revised: in which W is the Weiner process.

Original: The original CIR process is used to model the valuation of interest rate derivatives in ﬁnance.
Revised: The original CIR process is used to model the valuation of interest rate derivatives in ﬁnance.

Original: Equation (21) is solved by the forward Euler method with the time step size δt = 0 .
Revised: Equation (21) is solved by the forward Euler method with the time step size δt = 0 .

Original: The simulation is performed for T = (0 , 160000]
Revised: The simulation is performed for T = (0 , 160000]

Original: δt to generate the training data and T = (160000 , 162000]
Revised: δt to generate the training data and T = (160000 , 162000]

Original: δt is used for the testing.
Revised: δt is used for the testing.

Original: Note that the noise component of CIR is multiplicative, which depends on y ( t ) . 
Revised: Note that the noise component of CIR is multiplicative, which depends on y ( t ) . [Table]

Original: The experiments are performed for two different bin sizes, dy = 0 .
Revised: The experiments are performed for two different bin sizes, dy = 0 .

Original: 08 and 0 .
Revised: 08 and 0 .

Original: The DE-RNN has 64 LSTM cells.
Revised: The DE-RNN has 64 LSTM cells.

Original: Figure 1 shows the errors in the expectation and the standard deviation with respect to the analytical solution;
Revised: Figure 1 shows the errors in the expectation and the standard deviation with respect to the analytical solution;

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Here, p T denotes the true distribution of the CIR process.
Revised: Here, p T denotes the true distribution of the CIR process.

Original: The normalized root mean-squareerrors (NRMSE) are deﬁned as
Revised: The normalized root mean-square (NRMSE) are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: in which (cid:104)·(cid:105) denotes an average over the testing data, p L is the distribution from the LSTM, and sd [ y ] denotes the standard deviation of the data.
Revised: in which (cid:104)·(cid:105) denotes an average over the testing data, p L is the distribution from the LSTM, and sd [ y ] denotes the standard deviation of the data.

Original: The error in the expectation is normalized against a zeroth-order prediction, which assumes y t
Revised: The error in the expectation is normalized against a zeroth-order prediction, which assumes y t

Original: +1 = y t .
Revised: +1 = y t .

Original: In Figure 1, it is clearly shown that the prediction results are improved when a regularization is used to impose a smoothness condition.
Revised: In Figure 1, it is clearly shown that the prediction results are improved when a regularization is used to impose a smoothness condition.

Original: Comparing Figures 1 (a) and (b), for RCE, e µ and e σ become smaller when a smaller δy is used.
Revised: Comparing Figures 1 (a) and (b), for RCE, e µ and e σ become smaller when a smaller δy is used.

Original: As expected, e σ increases when λ is large.
Revised: As expected, e σ increases when λ is large.

Original: But, for the smaller bin size, δy = 0 . 04 , both e µ and e σ are not so sensitive to λ .
Revised: But, for the smaller bin size, δy = 0 . 04 , both e µ and e σ are not so sensitive to λ .

Original: Similar to RCE, e µ and e σ for CCE decrease at ﬁrst as the penalty parameter h increases.
Revised: Similar to RCE, e µ and e σ for CCE decrease at ﬁrst as the penalty parameter h increases.

Original: However, in general, RCE provides a better prediction compared to CCE.[Table]
Revised: However, in general, RCE provides a better prediction compared to CCE.

Original: NRMEs are listed in Table 1.
Revised: NRMEs are listed in Table 1.

Original: For a comparison, the predictions by AR(1) and KF are shown.
Revised: For a comparison, the predictions by AR(1) and KF are shown.

Original: The
Revised: The

Original: CIR process is essentially a ﬁrst-order autoregressive process.
Revised: CIR process is essentially a ﬁrst-order autoregressive process.

Original: So, it is not surprising to see that
Revised: So, it is not surprising to see that

Original: AR(1) and KF, which are designed for the ﬁrst-order AR process, outperforms DE-RNN for the prediction of the expectation.
Revised: AR(1) and KF, which are designed for the ﬁrst-order AR process, outperforms DE-RNN for the prediction of the expectation.

Original: However, e σ of AR(1) and KF are much larger than that of DE-RNN, because those models assume an additive noise. 
Revised: However, e σ of AR(1) and KF are much larger than that of DE-RNN, because those models assume an additive noise. The Gaussian process (GP) model has a relatively large e µ . But, GP outperforms AR(1) and KF in the prediction of the noise ( e σ ).

Original: Note that e σ of RCE and CCE are less than 4%, suggesting that DE-RNN can model the complex noise process very well.
Revised: Still, e σ of RCE and CCE are less than 4%, while that of GP is about 10%, indicating that DE-RNN can model the complex noise process much better.

Original: In Figure 2, a 200-step forecast by DE-RNN is compared with a Monte-Carlo solution of equation (21).
Revised: In Figure 2, a 200-step forecast by DE-RNN is compared with a Monte-Carlo solution of equation (21).

Original: DE-RNN is trained with δy = 0 . 04 and λ = 200 .
Revised: DE-RNN is trained with δy = 0 . 04 and λ = 200 .

Original: For the DE-RNN forecast, the testing data is supplied to DE-RNN for the ﬁrst 100 time steps, i.e., for t = − 10 to t = 0 , and the SMC multiple-step forecast is performed for the next 200 time steps with 20,000 samples.
Revised: For the DE-RNN forecast, the testing data is supplied to DE-RNN for the ﬁrst 100 time steps, i.e., for t = − 10 to t = 0 , and the SMC multiple-step forecast is performed for the next 200 time steps with 20,000 samples.

Original: It is shown that the multiple-step forecast by DE-RNN agrees very well with the MC solution of the CIR process.
Revised: It is shown that the multiple-step forecast by DE-RNN agrees very well with the MC solution of the CIR process.

Original: Note that, in Figure 2 (b), the noise process, as reﬂected in sd [ y t ] , is a function of y t , and hence the multi-step forecast of the noise increases rapidly ﬁrst and then decreases before reaching a plateau.
Revised: Note that, in Figure 2 (b), the noise process, as reﬂected in sd [ y t ] , is a function of y t , and hence the multi-step forecast of the noise increases rapidly ﬁrst and then decreases before reaching a plateau.

Original: The SMC forecast can accurately capture the behavior.
Revised: The SMC forecast can accurately capture the behavior.

Original: Such kind of behavior can not be represented if a simple additive noise is assumed.
Revised: Such kind of behavior can not be represented if a simple additive noise is assumed.

Original: 3. M ACKEY -G LASS TIME SERIES
Revised: 3. M ACKEY -G LASS TIME SERIES

Original: For the next test, we applied DE-RNN for a time series generated from the Mackey-Galss equation (Mackey & Glass, 1977);
Revised: For the next test, we applied DE-RNN for a time series generated from the Mackey-Galss equation (Mackey & Glass, 1977);

Original: [Equation]
Revised: [Equation]

Original: We use the parameters adopted from Gers (2001), α = 0 .
Revised: We use the parameters adopted from Gers (2001), α = 0 .

Original: 2 , β = 10 , γ = 0 .
Revised: 2 , β = 10 , γ = 0 .

Original: 1 , and τ = 17 .
Revised: 1 , and τ = 17 .

Original: The Mackey-Glass equation is solved by using a third-order Adams-Bashforth method with a time step size of 0.02.
Revised: The Mackey-Glass equation is solved by using a third-order Adams-Bashforth method with a time step size of 0.02.

Original: The time series is generated by down-sampling, such that the time interval between consecutive data is δt = 1 .
Revised: The time series is generated by down-sampling, such that the time interval between consecutive data is δt = 1 .

Original: A noisy observation is made by adding a white noise;
Revised: A noisy observation is made by adding a white noise;

Original: [Equation]
Revised: [Equation]

Original: (cid:15) t is a zero-mean Gaussian random variable with the noise level sd [ (cid:15) t ] = 0 .
Revised: (cid:15) t is a zero-mean Gaussian random variable with the noise level sd [ (cid:15) t ] = 0 .

Original: 3 sd
Revised: 3 sd

Original: [ y ] .
Revised: [ y ] .

Original: A time series of the length 1 . 6 × 10 5 δt is generated for the model trainig and another time series of length 2 × 10 3 δt is made for the validation.
Revised: A time series of the length 1 . 6 × 10 5 δt is generated for the model trainig and another time series of length 2 × 10 3 δt is made for the validation.

Original: DE-RNN is trained for δy = 0 .
Revised: DE-RNN is trained for δy = 0 .

Original: sd [ y ] and consists of 128 LSTM cells.
Revised: sd [ y ] and consists of 128 LSTM cells.

Original: Figure 3 (a) shows the noisy observation and the expectation of the next-step prediction, E [ˆ y t +1 | ˆ y t ] , in a phase space.
Revised: Figure 3 (a) shows the noisy observation and the expectation of the next-step prediction, E [ˆ y t +1 | ˆ y t ] , in a phase space.

Original: It is shown that DE-RNN can ﬁlter out the noise and reconstruct the original dynamics accurately.
Revised: It is shown that DE-RNN can ﬁlter out the noise and reconstruct the original dynamics accurately.

Original: Even though the noisy data are used as an input, E [ˆ y t +1 | ˆ y t ] accurately represents the original attractor of the chaotic system, indicating a strong de-noising capability of the LSTM. [Table]
Revised: Even though the noisy data are used as an input, E [ˆ y t +1 | ˆ y t ] accurately represents the original attractor of the chaotic system, indicating a strong de-noising capability of DE-RNN.

Original: The estimated probability distribution is shown in Figure 3 (b).
Revised: The estimated probability distribution is shown in Figure 3 (b).

Original: Without a regularization, the standard CE results in a noisy distribution, while the distribution from CCE shows a smooth Gaussian shape.
Revised: Without a regularization, the standard CE results in a noisy distribution, while the distribution from CCE shows a smooth Gaussian shape.

Original:  The prediction errors are shown in table 2.
Revised: [Table] The prediction errors are shown in table 2.

Original: NRMSEs are deﬁned as,
Revised: NRMSEs are deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: NRMSEs are computed with respect to the ground truth.
Revised: NRMSEs are computed with respect to the ground truth.

Original: Again, e µ compares the prediction error to the zeroth-order prediction.
Revised: Again, e µ compares the prediction error to the zeroth-order prediction.

Original: In this example, the errors are not so sensitive to the regularization parameters.
Revised: In this example, the errors are not so sensitive to the regularization parameters.

Original: The best result is achieved by CCE.
Revised: The best result is achieved by CCE.

Original: DE-RNN can make a very good estimation of the noise.
Revised: DE-RNN can make a very good estimation of the noise.

Original: The error in the noise component, e σ , is only 2% ∼ 5% .
Revised: The error in the noise component, e σ , is only 2% ∼ 5% .

Original: Unlike the CIR process, NRMSEs from KF and ARIMA are much larger than those of DE-RNN.
Revised: Unlike the CIR process, NRMSEs from KF and ARIMA are much larger than those of DE-RNN.

Original: Because the underlying process is a delay-time nonlinear dynamical system, those linear models can not accurately approximate the complex dynamics.
Revised: Because the underlying process is a delay-time nonlinear dynamical system, those linear models can not accurately approximate the complex dynamics.

Revised: Since GP is capable of representing a nonlinear behavior of data, GP outperforms KF and ARIMA both in e µ and e σ .

Revised: In particular, e σ of GP is similar to that of DE-RNN.

Revised: However, e µ of GP is about 1.5 times larger than DE-RNN.

Original: A multiple-step forecast of the Mackey-Glass time series is shown in Figure 4.
Revised: A multiple-step forecast of the Mackey-Glass time series is shown in Figure 4.

Original: In the validation time series, the observations in t ∈ [1 , 100] δt are supplied to the DE-RNN to develop the internal state, and a 500-step forecast is made for t ∈ [101 , 600] δt .
Revised: In the validation time series, the observations in t ∈ [1 , 100] δt are supplied to the DE-RNN to develop the internal state, and a 500-step forecast is made for t ∈ [101 , 600] δt .

Original: In Figure 4 (a), it is shown that a multiple-step forecast by a standard regression LSTM approximates y ( t ) very well initially, e.g, for t < 80 δt , but eventually diverges for larger t .
Revised: In Figure 4 (a), it is shown that a multiple-step forecast by a standard regression LSTM approximates y ( t ) very well initially, e.g, for t < 80 δt , but eventually diverges for larger t .

Original: Because of the Mackey-Glass time series is chaotic, theoretically it is impossible to make a long time forecast.
Revised: Because of the Mackey-Glass time series is chaotic, theoretically it is impossible to make a long time forecast.

Original: But, in the DE-RNN forecast, y ( t ) is bounded by the 95%-conﬁdence interval (CI) even for the 500-step forecast.
Revised: But, in the DE-RNN forecast, y ( t ) is bounded by the 95%-conﬁdence interval (CI) even for the 500-step forecast.

Original: Note that the uncertainty, denoted by 95%-CI grows only at a very mild rate in time.
Revised: Note that the uncertainty, denoted by 95%-CI grows only at a very mild rate in time.

Original: In fact, it is observed that CI is not a monotonic function of time.
Revised: In fact, it is observed that CI is not a monotonic function of time.

Original: In DE-RNN, the 95%-CI may grow or decrease following the dynamics of the system, while for the conventional time series models, such as ARIMA and KF, the uncertainty is a non-decreasing function of time.
Revised: In DE-RNN, the 95%-CI may grow or decrease following the dynamics of the system, while for the conventional time series models, such as ARIMA and KF, the uncertainty is a non-decreasing function of time.

Original: 3. M AUNA L OA CO 2 OBSERVATION
Revised: 3. M AUNA L OA CO 2 OBSERVATION

Original: In this experiments, DE-RNN is tested against the atmospheric CO 2 observation at Mauna Loa Observatory, Hawaii (Keeling et al., 2001).
Revised: In this experiments, DE-RNN is tested against the atmospheric CO 2 observation at Mauna Loa Observatory, Hawaii (Keeling et al., 2001).

Original: The CO 2 data set consists of weekly-average atmospheric
Revised: The CO 2 data set consists of weekly-average atmospheric

Original: CO 2 concentration from Mar-29-1958 to Sep-23-2017 (Figure 5 a).
Revised: CO 2 concentration from Mar-29-1958 to Sep-23-2017 (Figure 5 a).

Original: The data from Mar-29-1958 to
Revised: The data from Mar-29-1958 to

Original: Apr-01-2000 is used to train DE-RNN and a 17-year forecast is made from Apr-01-2000 to Sep-232017.
Revised: Apr-01-2000 is used to train DE-RNN and a 17-year forecast is made from Apr-01-2000 to Sep-232017.

Original: This CO2 data has been used in previous studies (Gal & Ghahramani, 2016; Rasmussen & Williams, 2006). In DE-RNN, 64 LSTM cells and δy = 0 .
Revised: This CO

Original: 1 sd [ dy t ] , in which dy t = y t +1 − y t , are used.
Revised: [Table]

Original: The 17-year DE-RNN forecast, with 1,000 MC samples, is shown in Figure 5 (b).
Revised: The 17-year DE-RNN forecast, with 1,000 MC samples, is shown in Figure 5 (b).

Original: DE-RNN well represents the growing trend and the oscillatory patten of the CO 2 data.
Revised: DE-RNN well represents the growing trend and the oscillatory patten of the CO 2 data.

Original: The CO 2 data is nonstationary, where the rate of increase of CO 2 is an increasing function of time.
Revised: The CO 2 data is nonstationary, where the rate of increase of CO 2 is an increasing function of time.

Original: Since DE-RNN is trained against the history data, where the rate of CO 2 increase is smaller than the current, it is expected that the forecast will underestimate the future CO 2 .
Revised: Since DE-RNN is trained against the history data, where the rate of CO 2 increase is smaller than the current, it is expected that the forecast will underestimate the future CO 2 .

Original: E [ˆ y n + s | ˆ y s ] agrees very well with the observation for the ﬁrst 200 weeks, but eventually underestimates CO 2 concentration.
Revised: E [ˆ y n + s | ˆ y s ] agrees very well with the observation for the ﬁrst 200 weeks but eventually underestimates CO 2 concentration.

Original: It is interesting to observe that the upper bound of the 95%-CI grows more rapidly than the expectation and provides a good approximation of the observation.
Revised: It is interesting to observe that the upper bound of the 95%-CI grows more rapidly than the expectation and provides a good approximation of the observation.

Original: For a comparison, the forecast by a regression
Revised: For a comparison, the forecast by a regression

Original: LSTM is also shown.
Revised: LSTM is also shown.

Original: Similar to the chaotic Mackey-Glass time series, the regression LSTM makes a good prediction for a short time, e.g., t < 100 weeks, but eventually diverges from the observation.
Revised: Similar to the chaotic Mackey-Glass time series, the regression LSTM makes a good prediction for a short time, e.g., t < 100 weeks, but eventually diverges from the observation.

Original: Note that the lower bound of 95%-CI encompasses the regression LSTM.
Revised: Note that the lower bound of 95%-CI encompasses the regression LSTM.

Original: 3. CPU TEMPERATURE FORECAST
Revised: Figure 5 (c) shows a forecast made by GP, following setup suggested by Rasmussen & Williams (2005). For a shortterm forecast ( < 50 weeks), GP provides a sharper estimate of the uncertainty, i.e., a smaller 95%- CI interval. However, even for a mid-term forecast, 100 ∼ 600 weeks, the ground truth is close or slightly above the upper bound of 95%-CI.

Revised: Because of the different behaviors, it is difﬁcult to conclude which method, DE-RNN or GP, provides a better forecast.

Revised: But, it should be noted that the GP is based on a combination of handcrafted kernels speciﬁcally designed for this particular problem, while such careful tuning is not required for DE-RNN.

Original: In the last experiment, IBM Power System S822LC and NAS Parallel Benchmark (NPB) are used to generate the temperature trace.
Revised: In the last experiment, IBM Power System S822LC and NAS Parallel Benchmark (NPB) are used to generate the temperature trace.

Original: Figure 6 (a) shows the temperature of a CPU.
Revised: Figure 6 (a) shows the temperature of a CPU.

Original: The temperature sensor generates a discrete data, which has a resolution of 1 ◦
Revised: The temperature sensor generates a discrete data, which has a resolution of 1 ◦

Original: C. The CPU temperature is controlled by three major parameters; CPU frequency, CPU utilization, and cooling fan speed.
Revised: C. The CPU temperature is controlled by three major parameters; CPU frequency, CPU utilization, and cooling fan speed.

Original: In this experiment, we have randomized the CPU frequencies and job arrival time to mimic the real workload behavior, while the fan speed is ﬁxed to 3300RPM.
Revised: In this experiment, we have randomized the CPU frequencies and job arrival time to mimic the real workload behavior, while the fan speed is ﬁxed to 3300RPM.

Original: The time step size is δt = 2 seconds.
Revised: The time step size is δt = 2 seconds.

Original: Accurate forecast of CPU temperature for a future workload scenario is essential in developing an energy-efﬁcient control strategy for the thermal management of a cloud system.
Revised: Accurate forecast of CPU temperature for a future workload scenario is essential in developing an energy-efﬁcient control strategy for the thermal management of a cloud system.

Original: Figure 6 (c) and (d) show multiple-step forecasts of the CPU temperature by RCE and a regression LSTM, respectively.
Revised: Figure 6 (c) and (d) show multiple-step forecasts of the CPU temperature by RCE and a regression LSTM, respectively.

Original: The bin size is δy = 0 .
Revised: The bin size is δy = 0 .

Original: 18 ◦ C, which is smaller than the sensor resolution.
Revised: 18 ◦ C, which is smaller than the sensor resolution.

Original: In the forecast, the future control parameters are given to DE-RNN.
Revised: In the forecast, the future control parameters are given to DE-RNN.

Original: In other words, DE-RNN[Table] predicts the probability distribution of future temperature with respect to a control scenario, i.e., p (ˆ y t + n | (cid:98)
Revised: In other words, DE-RNN predicts the probability distribution of future temperature with respect to a control scenario, i.e., p (ˆ y t + n | (cid:98)

Original: Y 0: t , U 0: t + n − 1 ) .
Revised: Y 0: t , U 0: t + n − 1 ) .

Original: The forecast is made by using 5,000 Monte Carlo samples.
Revised: The forecast is made by using 5,000 Monte Carlo samples.

Original: Here, 1800-step forecast is made, t =∼ 3 , 600 sec.
Revised: Here, 1800-step forecast is made, t =∼ 3 , 600 sec.

Original: and only the results in t ∈ (50 , 1800) sec. is shown.
Revised: and only the results in t ∈ (50 , 1800) sec. is shown.

Original: While the regression LSTM makes a very noisy prediction near the local peak temperature at t (cid:39) 500 ,
Revised: While the regression LSTM makes a very noisy prediction near the local peak temperature at t (cid:39) 500 ,

Original: RCE provides a much more stable forecast.
Revised: RCE provides a much more stable forecast.

Original: Table 3 shows the l ∞ -errors, i.e., maximum absolute difference.
Revised: Table 3 shows the l ∞ -errors, i.e., maximum absolute difference.

Original: The maximum error is observed near the peak temperature at t (cid:39) 500 .
Revised: The maximum error is observed near the peak temperature at t (cid:39) 500 .

Original: ARIMA and KF are also tested for the multiple-step forecast, but the results are not shown because their performance is much worse than the LSTMs.
Revised: ARIMA, KF, and GP are also tested for the multiple-step forecast, but the results are not shown because their performance is much worse than the LSTMs.

Original: The changes in the temperature are associated with step changes of some control parameters.
Revised: The changes in the temperature are associated with step changes of some control parameters.

Original: Such abrupt transitions seem to cause the large oscillation in the regression LSTM prediction.
Revised: Such abrupt transitions seem to cause the large oscillation in the regression LSTM prediction.

Original: But, for RCE or CCE, the prediction is made from an ensemble of Monte Carlo samples, which makes it more robust to such abrupt changes.
Revised: But, for RCE or CCE, the prediction is made from an ensemble of Monte Carlo samples, which makes it more robust to such abrupt changes.

Original: Also, note that for t < 200 sec., 
Revised: Also, note that for t < 200 sec., RCE prediction ( (cid:39) 53 .

Original: RCE prediction ( (cid:39) 53 . 4 ◦ C) is in between the two discrete integer levels, 53 ◦ C and 54 ◦ C, which correctly reﬂects the uncertainty in the measurement, while the regression LSTM ( (cid:39) 53 .
Revised: 4 ◦ C) is in between the two discrete integer levels, 53 ◦ C and 54 ◦ C, which correctly reﬂects the uncertainty in the measurement, while the regression LSTM ( (cid:39) 53 .

Original: 1 ◦ C) more closely follows one of the two levels. 
Revised: 1 ◦ C) more closely follows one of the two levels. 3. L ORENZ T IME S ERIES

Revised: Finally, in this experiment we evaluate the performance of the multivariate DE-RNN, for which we used a noisy multivariate time series generated by the Lorenz equations (Lorenz, 1963)

Original: C ONCLUDING REMARKS
Revised: [Equation] [Equation] We used the coefﬁcients from Lorenz (1963), which are α 1 = 10 , α 2 = 8 / 3 , and α 3 = 28 . The system of equations (27) is solved by using a third-order Adams-Bashforth method with a time step size of 0.001 and a time series data set is generated by downsampling, such that δt = 0 .

Revised: A multivariate joint normal distribution is added to the ground truth to generate a noisy time series, i.e.,

Revised: [Equation]

Revised: Here, the noise level is set to σ 1 = 0 .

Revised: 2 sd [ y (1) ] , σ 2 = 0 .

Revised: 3 sd [ y (2) ] , and σ 3 = 0 .

Revised: 2 sd [ y (3) ] .

Revised: Three

Revised: DE-RNNs are trained to model p (ˆ y (1) t +1 |

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: ˆ

Revised: [Equation]

Revised: ˆ

Revised: [Equation]

Revised: ˆ

Revised: [Equation]

Revised: re

Revised: [Equation]

Revised: [Table]

Revised: spectively, with the grid spaces δy ( i )

Revised: = 0 .

Revised: 02 sd [ˆ y ( i ) ]

Revised: for i = 1 , 2 , 3 .

Revised: Each DE-RNN has 128 LSTM cells.

Revised: The penalty parameters for RCE and CCE are λ = 50 and h = 5 , respectively.

Revised: Figure 4 shows normalized errors for the next-step prediction.

Revised: The normalized errors are deﬁned as e µ =

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ˆ y ( j ) t +1 ]

Revised: Σ ij − 1 .

Revised: The moments of the joint PDF are computed by the Monte Carlo method with a sample size of 5 × 10 3 .

Revised: It is shown that DE-RNN makes a very good prediction of both expectations and covariances.

Revised: The error in the covariance is less than 4% except for Σ 12 .

Revised: It is shown that DE-RNN is able to make correct predictions of not only the magnitude, but also the signs of the covariance.

Revised: A vector autoregressive model (VAR) also predicts the signs of the covariance correctly.

Revised: But, the errors in the expectation and covariances are much larger than DE-RNN.

Original: {+The GP used in the experiment assumes an independent noise, i.e. Σ = ρ 
+}
Revised: {+The GP used in the experiment assumes an independent noise, i.e. Σ = ρ 
+}

Revised: I .

Revised: Hence, e Σ ij is not evaluated for GP.

Revised: Similar to the MackeyGlass time series, GP outperforms VAR, but the errors are larger than DE-RNN.

Revised: The error in e σ is about 10 times larger than DE-RNN, while e µ of GP is about 3 times larger than DE-RNN.

Revised: Figure 7 shows 300-step forecasts of the Lorenz time series.

Revised: The expectations from DE-RNNs make a good prediction of the ground truth up to t − s < 1 . 5 .

Revised: Then, the expectations start to diverge from the ground truth, which is accompanied by a sudden increase in the 95%-CI.

Revised: For a longer-time forecast, e.g., t − s > 4 , the 95%-CI exhibits an oscillatory patten depending on the dynamics of the

Revised: Lorenz system and well captures the oscillations in the ground truth.

Original: We present DE-RNN to compute the time evolution of a probability distribution for complex time series data.
Revised: We present DE-RNN to compute the time evolution of a probability distribution for complex time series data.

Original: DE-RNN employs LSTM to learn multiscale, nonlinear dynamics from the noisy observations, which is supplemented by a softmax layer to approximate a probability density function.
Revised: DE-RNN employs LSTM to learn multiscale, nonlinear dynamics from the noisy observations, which is supplemented by a softmax layer to approximate a probability density function.

Original: To assign probability to the softmax output, we use a mapping from R to N + , which leads to a cross-entropy minimization problem.
Revised: To assign probability to the softmax output, we use a mapping from R to N + , which leads to a cross-entropy minimization problem.

Original: To impose a geometric structure in the distribution, two regularization strategies are proposed.
Revised: To impose a geometric structure in the distribution, two regularization strategies are proposed.

Original: The regularized cross-entropy method is analogous to the penalized maximum likelihood estimate for the density estimation, while the convolution cross-entropy method is motivated by the kernel density estimation.
Revised: The regularized cross-entropy method is analogous to the penalized maximum likelihood estimate for the density estimation, while the convolution cross-entropy method is motivated by the kernel density estimation.

Original: The proposed algorithm is validated againsttwo synthetic data set, for which we can compare with the analytical solutions, and two real data sets. 
Revised: The proposed algorithm is validated against data , for which we can compare with the analytical solutions, and two real data sets. A PPENDIX A M ULTIVARIATE DE-RNN Recall the product rule from Section 2.

Original: In this study, for simplicity, the problem is formulated fora univariate  time series.
Revised: [Equation] The extension of DE-RNN algorithm for univariate data to an l -dimensional multivariate time series is straightforward. First, deﬁne the discretization grid points, α ( i ) = ( α ( i ) 0 , · · · , α ( i ) K i ) , for every variable, i.e., i = 1 , · · · , l .

Original: But, it isstraightforward to extend the methodology to a multivariate time series.
Revised: Here, K i is the number of the discretization intervals for the i -th variable. Then, we can deﬁne the discretization intervals, I ( i ) j = ( α ( i ) j − 1 , α ( i ) j ) , and the mapping functions, C ( i ) ( y ( i ) ) , for the l variables.

Revised: The ﬁrst component of the product rule is the marginal PDF, p (ˆ y (1) t +1 ) .

Revised: Hereafter, the obvious dependency on (cid:98)

Revised: Y 0: t , U 0: t in the notation is omitted for simplicity.

Revised: The marginal PDF can be computed by the same method as for the univariate time series.

Revised: To train DE-RNNs for the conditional PDFs for the i -th variable, p (ˆ y ( i ) t +1 | ˆ y ( i − 1) t +1 , · · · ˆ y (1) t +1 ) , the original time series data of length

Revised: N , D R

Revised: =

Revised: ˆ y t ; ˆ y t ∈ R l , and t = 1 , . .

Revised: , N } , is discretized by using C ( i ) , which gives us

Revised: D ( i ) C

Revised: = { ( c ( i ) t , ˆ y t ); c ( i ) t ∈ N + , ˆ y t ∈ R l , and t = 1 , . . .

Revised: , N } , where c ( i ) t

Revised: = C ( i ) (ˆ y ( i ) t ) .

Revised: Then the output softmax layer, P ( i ) t +1 , is computed by an LSTM as

Revised: [Equation]

Revised: 30) in which s ( i ) t is the internal state of the i -th variable.

Revised: In other words, in the training of the DE-RNN for p (ˆ y ( i ) t +1 | ˆ y

Revised: ( i − 1) t +1 , · · · ˆ y (1) t +1 ) , the input vector is the variables at the current time step, ˆ y t , combined with the conditioning variables in the next step, (ˆ y (1) t +1 , · · · , ˆ y ( i − 1) t +1 ) , and the target is the class label, c ( i ) t +1 , in the next time step.

Revised: The DE-RNN can be trained by minimizing RCE or CCE as described in

Revised: Section 2.2.

Revised: Observe that during the training phase, each DE-RNN is independent from each other.

Revised: Therefore, all the DE-RNNs can be trained in parallel, signiﬁcantly improving the computational efﬁciency, enabling the method to scale only linearly with the number of dimensions l .

Revised: Once all the conditional PDFs are obtained, the joint PDF can be computed by a product of the

Revised: DE-RNN outputs.

Revised: For a demonstration, the covariance of a bivariate time series can be computed as

Original: A A PENDIX
Revised: [Equation] where the time index, ( t +1 ), is omitted in the notation of the softmax output, P ( i ) j , and the subscript j denotes the j -th element of the softmax layer. Note that, although there is no dependency between DE-RNNs during training, in the prediction phase of computing the joint PDF, there is a hierarchical dependency between all the DE-RNNs.

Revised: This kind of direct numerical integration does not scale well for number of dimensions l (cid:29) 1 .

Revised: For a high dimensional system, a sparse grid (Xiu & Hesthaven, 2005) or a Monte Carlo method can be used to speed up the numerical integration.

Revised: We outline a

Revised: Monte Carlo procedure in Algorithm 2.

Revised: Comparing with Algorithm 1, an extension of Algorithmfor a multiple-step forecast of a multivariate time series is straightforward.

Original: In this Section we present a few extra details of the proposed DE-RNN algorithm.
Revised: In this Section we present a few extra details of the proposed DE-RNN algorithm.

Original: Figure 7 shows the architecture of DE-RNN as was used during the experiments(presented in Section 3).
Revised: Figure 8 shows the architecture of DE-RNN as was used during the experiments in Section 3.

Original: In Figure 8 we show the process of computing one-step-ahead predictions (presented in Section 2.2).
Revised: In Figure 9 we show the process of computing one-step-ahead predictions of univariate time series as was presented in Section 2.2.

Original: Note that since DE-RNN estimates approximation of the predictive probability distribution, the predicted value, e.g., for time step t + 1 , is the discrete approximation of E [ˆ y t +1 | (cid:98)
Revised: Note that since DE-RNN estimates approximation of the predictive probability distribution, the predicted value, e.g., for time step t +1 , is the discrete approximation of E [ˆ y t +1 | (cid:98)

Original: Y 0: t , U 0: t ] , i.e., the expectation of ˆ y t
Revised: Y 0: t , U 0: t ] , i.e., the expectation of ˆ y t

Original: +1 given all the observations and control inputs up to time t . 
Revised: +1 given all the observations and control inputs up to time t . Finally, in

Revised: Input : (cid:98)

Revised: Y 0: t , and number of Monte Carlo samples, N s

Revised: Output : p ( ˆ y t +1 | (cid:98)

Original: Finally, in Figure 9 we show the details of  the multi-step forecast (presented in Section 2.3, in Algorithm 1).
Revised: Y 0: t ) (density estimation from ˆ y t +1 ) Initialization: Set LSTM states to s 0 = h 0 = Perform a sequential update of LSTM up to time t − 1 from the noisy observations ( (cid:98) Y 0: t ): for i = 1 , t − 1 do s (1) i =

Revised: Ψ e ( ˆ y i , s (1) i − 1 ) for j = 2 , l do s ( j ) i =

Revised: Ψ e (ˆ y (1) i +1 , · · · , ˆ y ( j − 1) i +1 , ˆ y i , s ( j ) i − 1 ) end for end for

Revised: Monte Carlo sampling for p ( ˆ y t +1 | (cid:98)

Revised: Y 0: t ) :

Original: Using sequential Monte Carlo method, the discrete approximation of the predictive distribution p (ˆ y  t +  n | (cid:98)
Revised: Compute the predictive distribution of ˆ y (1) t + [Equation] for n = 1 , N s do Draw a sample, ˆ y (1) ,n t +1 , from P (1) t +1 .

Original: Y 0: t , U 0: t + n − 1 ) is estimated using N s samples.
Revised: for j = 2 , l do Compute the conditional distribution for ˆ y ( j ) t +1 for each sample

Original: [Figure]
Revised: [Equation]

Revised: Draw a sample, ˆ y ( j ) ,n t +1 , from P ( j ) ,n t + end for end for

Revised: Figure 10 we show the details of the multi-step forecast for univariate time series as was presented in Algorithm 1, in Section 2.4.

Revised: Using sequential Monte Carlo method, the discrete approximation of the predictive distribution p (ˆ y t + n |

Original:  [Figure] 
Revised: [Equation] [Table] [Figure] L

Original: u
Revised: L L softplus softmax LSTM P t

Original: [Figure]
Revised: [Figure]

Original: +2 | b
Revised: DE

Original: Y 0: t +1 , U 0: t +⇤
Revised: [Figure] RNN z t + [Figure] h t +

Original: E ⇥ ˆ y t +1 | b
Revised: [Figure] discrete approximation of predictive distribution p (ˆ y t +1 | b

Original: Y 0: t , U 0: t ⇤
Revised: Y 0: t , U 0: t )

Original:  ˆ y 1 t + u t sample over all bins (using ) sample within selected bin (uniformly) ↵↵ K
Revised: [Figure] ˆ y 1 t + u t sample over all bins (using ) sample within selected bin (uniformly) ↵↵ K

Original: P 1 t
Revised: P 1 t

Original: + ˆ y t s 1 t h 1 t 
Revised: + ˆ y t s 1 t h 1 t

Original: DE - RNN ↵
Revised: DE - RNN ↵

Original: DE - RNN ˆ y 1 t + ˆ y 1 t +1 u t + h 1 t s 1 t
Revised: DE - RNN ˆ y 1 t + ˆ y 1 t +1 u t + h 1 t s 1 t

Original: P 1 t + ↵ K ↵↵ K
Revised: P 1 t + ↵ K ↵↵ K

Original: DE - RNN ˆ y 1 t +
Revised: DE - RNN ˆ y 1 t +

Original: P 1 t + s 1 t + h 1 t + ˆ y 1 t +2 u t +
Revised: P 1 t + s 1 t + h 1 t + ˆ y 1 t +2 u t +

Original: N s  ˆ y N s t
Revised: N s  ˆ y N s t

Original: + ...
Revised: + ...

Original: ˆ y 3 t + ˆ y 2 t
Revised: ˆ y 3 t + ˆ y 2 t

Original: + ˆ y 1 t +   density estimation
Revised: + ˆ y 1 t +   density estimation

Original: P 1 t + replicate hidden states times p (ˆ y t +3 | b
Revised: P 1 t + replicate hidden states times p (ˆ y t +3 | b

Original: Y 0: t , U 0: t +2 )
Revised: Y 0: t , U 0: t +2 )