N EURAL M ARKOV C ONTROLLED SDE: S TOCHASTIC O PTIMIZATION FOR C ONTINUOUS -T IME D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel probabilistic framework for modeling stochastic dynamics with the rigorous use of stochastic optimal control theory.
The proposed model called the neural Markov controlled stochastic differential equation (CSDE) overcomes the fundamental and structural limitations of conventional dynamical models by introducing the following two components: (1) Markov dynamic programming to efﬁciently train the proposed CSDE and (2) multi-conditional forward-backward losses to provide information for accurate inference and to assure theoretical optimality.
We demonstrate that our dynamical model efﬁciently generates a complex time series in the data space without extra networks while showing comparable performance against existing model-based methods on several datasets.
I NTRODUCTION
Recently, there has been interest in using continuous dynamical systems to approximate complex time series.
Neural ODE Chen et al.
(2018), which opened the way for continuous representation of neural networks, have been widely investigated and thoroughly analyzed by Massaroli et al.
As the stochastic generalization of ODE, Neural SDEs Li et al.
(2020) have been proposed by regarding intrinsic stochasticity in data representations ( e .
g ., stock market data).
Since the conventional Neural
ODE/SDEs only utilize the initial information of trajectories when propagating dynamics, modelling complex time-series with naive Neural ODE/SDEs has been regarded as inefﬁcient and undesirable choices, as pointed out by Kidger et al.
To address these problems, Rubanova et al.
(2019) presented an auto-regressive model to generalize recurrent neural networks (RNNs) to have continuous hidden dynamics with neural ODE.
Furthermore,
Chen et al.
(2018) proposed an encoder-decoder structure with Neural ODE in the latent space to reconstruct/predict complex data representation.
Although the aforementioned approaches produce remarkable results, they focus on suggesting additional probabilistic structures rather than improving the learnability of the Neural ODE model itself.
Compared to aforementioned approaches, we focus on solving the fundamental issues of Neural ODE/SDEs.
First, we raise two important questions.
Q1)
How can we construct an efﬁcient network architecture for Neural ODE/SDE models that do not require additional recurrent networks to model complex time series?
Q2)
How can we train Neural ODE/SDEs that can utilize richer information of observed sequences to accurately generate complex time series?
As SDEs can be posed as stochastic generalizations of ODEs, we focus on a stochastic framework and adopt the stochastic optimal control theory as our primary analysis tool for the rigorous and systematic analysis of the aforementioned problems.
Keeping this in mind, the contributions of our paper are to answer the above two questions.
A1)
Novel probabilistic framework for stochastic dynamics.
We propose a novel neural controlled stochastic differential equation (CSDE) to model the complex stochastic time series where multiple control agents are deﬁned to construct local dynamics in their own private temporal states.
With this property, the proposed CSDE incorporates Markov dynamic programming, enables our model to directly infer the complex trajectory on data space rather than the latent space without any extra network ( e .
g ., encoders/decoders), and shows remarkable efﬁciency compared to existing methods.
A2)
Novel conditional losses.
We introduce a novel Markov forward conditional (MFcond) loss to utilize multi-conditioned dynamics instead of the conventional dynamics determined by partial initial conditions.
The proposed MFcond loss makes our method to model the complex information of time-sequential data.
To impose regularization and to ensure the optimality of control agents, we also suggest a novel Markov backward conditional (MBcond) loss.
R ELATED W ORK
ODE As a Latent Probabilistic Model.
Rubanova et al.
(2019) suggested an ODE-RNN by combining RNN with the latent dynamics induced by the Neural ODE.
To deal with irregular time-stamps, exponential-decaying of the hidden states was also discussed by Che et al.
De Brouwer et al.
(2019) assumed that the observations are sampled from the stochastic dynamics induced from SDEs and introduced GRU-ODE to approximate the observed stochastic time series.
SDE As a Latent Probabilistic Model.
Liu et al.
(2021) incorporated Neural SDEs with recurrent models as a primary probabilistic dynamical model to generate stochastic continuous-time latent variables.
While this SDE model could describe the stochastic dynamics on the latent space with recurrent structures ( e .
g ., RNN encoder/decoder), it required a whole sequence of historical observations as inputs to the model.
Unfortunately, this type of formulation leads to non-Markov types of
SDEs, which makes it difﬁcult to analyze the probabilistic characteristics of the dynamics.
Unlike this model, we focus on the Markov SDEs while maintaining identical objectives.
Neural CDE and RDE.
Kidger et al.
(2020) proposed a data-driven neural controlled differential equation called Neural CDE to incorporate a rough-path analysis theory and model complex time series.
Morrill et al.
(2021) extended the rough-path theory with a Neural RDE to deal with the continuous time series over long time.
Generative SDE Models.
Recently, Kidger et al.
(2021) suggested SDE-based generative adversarial networks (GANs).
Park et al.
(2021) utilized the temporal conditional Wasserstein distance to construct GANs for time-series generation.
Please refer to Appendix A.1 for additional discussion on related works.
M ARKOV N EURAL C ONTROLLED SDE
In Section 3.1, we introduce a novel SDE model that considers temporally private agents.
In
Section 3.2, we propose the Markov-DP-TP framework to efﬁciently solve the stochastic optimal control problem with the proposed neural SDE model.
Finally, we suggest novel Markov conditional forward and backward losses in Section 3.3 and 3.4, respectively.
In the Appendix, we provided the detailed technical deﬁnitions.
3. C ONTROLLED S TOCHASTIC D IFFERENTIAL E QUATIONS
The basic object of our interest is a controlled F t -adapted process X αt with multiple control agents
α = { α 1 , · · · , α M } ∈ A where A denotes the set of admissible control agents.
In particular, the stochastic process X αt is deﬁned as a solution to the following CSDE:
[Equation]
where b and σ : [0 , T ] × R d × A → R d are the drift and diffusion functions, respectively.
Each control agent α i : [0 , T ] × R d × R m , α i = α
i ( t, X t ; θ i ) , ∀ 1 ≤ i ≤ M is deﬁned as a Markov closed-loop feedback control, which is parameterized by the neural network θ i .
While every agent is deﬁned as a closed-loop feedback-type Carmona (2016b), the solution to the CSDE above, X αt , is the Markov process , which means that process X αt is propagated using the information of the current state.
Let T
=
t k } 1 ≤ k ≤ N be a set of ordered times 1 such that
= t 1 < · · · < t k < t l < · · · < t N = T .
The set of functions { w i ( t ) } 1 ≤ i ≤ M is deﬁned as an indicator function on the intervals, w i ( t ) = 1 t k ≤ t ≤ t l with predetermined starting/ending points t k , t l in T .
We call this function temporal privacy (TP) because it represents each agent’s attention on different sub-intervals.
Overall, in (1), the stochastic process X α t is propagated by summing M -number of individual agent’s weighted attentions (cid:110)(cid:80) M w i
b i ( · , · , α i ) , (cid:80)
M w i σ i ( · , · , α i ) (cid:111) .
To understand the behavior of the proposed
CSDE more deeply, we consider the following detailed example:
Role of Temporal Privacy.
We deﬁne w r ( s ) = 1 t ≤ s ≤ u , t, u ∈ T with r ≤ M .
Then, X αu in (1) given X t at an interval [ t, u ] can be equivalently rewritten in the integration form:
X α
[Equation]
[ α 1 ,
[Equation]
u [Equation] In (2), the activated control agent to evaluate the stochastic process X αu for the interval [ t, u ] is only α r ( i .
e .,
X αu =
X α r u ) owing to the deﬁnition of the weighting function w ( · ) ( t ) .
This means that the remaining control agents { α j } j (cid:54) =
r are not used for the evaluation of the stochastic process in the sub-interval [ t, u ] .
While each agent α i is activated at its own private sub-interval, this leads our method to adopt dynamic programming (DP) to train Neural CSDEs in the form of (1).
In this paper, we aim to solve the optimal control problem via DP with multiple agents, where each agent specializes in solving a particular sub-problem in its private interval.
3. M ARKOV D YNAMIC P ROGRAMMING P RINCIPLES
The dynamic programming principle is one of the fundamental philosophies for dealing with stochastic optimal control problems.
Its basic idea is to consider a family of sub-problems with different initial times/states and establish the relation among the sub-problems to systemically solve them.
Using the mathematical property of the proposed CSDE with TP, we present an efﬁcient learning strategy to solve stochastic optimal control problems via Markov dynamic programming (Markov-DP).
In this paper, we aim to solve the stochastic optimal control problem by training control agents α
=
[ α 1 , · · · , α M ] and minimizing the cost functional J ( t, X αt ) : [ [Equation] T ] × R d → R + :
[Equation]
where l : [0 , T ] × R d → R + is the running cost ( e .
g ., L 2 loss) that computes the discrepancy between the propagated process X αt and the observed data point y t at each time t , Ψ( X αT ) :
R d → R + is the terminal cost that estimates the discrepancy between the terminal state and the data y T .
To evaluate the cost functional J ( t, X αt ) at time t with control agents α , the running cost is integrated over the time interval [ t, T ] conditioned on ﬁltration F t .
Note that the expectation conditioned on F t in (3) can be substituted to the expectation conditioned on X αt in light of the Markov property presented in
Section A.2, and the cost functional at time t only depends on the current state of the process X αt .
Markov-DP with Temporal Privacy.
By combining the tower property of the conditional expectations with the dynamic programming principle and It ˆ o’s formula (Oksendal (1992)), one can show that a minimization problem can be recursively decomposed into sub-problems owing to the property of TP in our proposed CSDE:
[Equation]
where V is an optimal cost functional ( i .
e ., value function), α r denotes the r -th control agent, and α ( − r ) =
[ α 1 , · · · , 0 , · · · , α M ] indicates the set of remaining agents (the r -th component is zero).
In (4), the minimization problem (A) over α is divided into two sub-problems using the dynamic programming principle, which are (B) and (B’) .
Because the minimization problem (B) is only dependent on the control agent α r parameterized by the neural network θ r , we compute the gradient descent of θ r to solve the sub-problem (B) :
[Equation]
where w r ( s )
=
1 t ≤ s ≤ u is the TP function at an interval [ t, u ] and k is the index for the learning iterations.
In (5), the r -th control agent α r minimizes the cost functional using the gradient descent scheme at its own temporal sub-interval.
As the remaining sub-problem (B’) over agents α ( − r ) can also be recursively decomposed into smaller sub-problems using the dynamic programming principle, the original problem (A) is solved separately with M -number of control agents
α = { α 1 , · · · , α M } with the M -number of gradient descent schemes.
This indicates that we can obtain the set of agents α (cid:63) =
{ α i ( · , · ; θ i(cid:63) ) } by collecting individual optimal agents with sub-problems.
In this paper, we combine the Markov-DP with M gradient descent schemes in (5) and CSDE with TP in (1) and introduce a novel Markov-DP-TP framework .
In the numerical experiments in
Section 4.4, we show that the proposed Markov-DP-TP framework remarkably increases the model efﬁciency compared to conventional non-DP naive approaches, which makes our method directly model the complex time series in the data space.
However, despite the improvements with our novel
Markov-DP-TP framework, there exist remaining practical/theoretical issues that should be addressed to solve the optimal control problem with complex datasets.
1) Conditional Dependency.
The main practical issue in implementing the Markov-DP-TP framework is that explicit conditional states are not given, e
g ., X αt in (5).
As different initial/terminal conditions of SDE lead to totally different behaviors of induced dynamics, well-designed conditional information is a crucial factor in training the Neural CSDE for speciﬁc applications.
In Section 3.3, we introduce the Markov Forward conditional (MFcond) loss to train the Neural CSDE with well-posed conditional information that ensures accurate network predictions.
2) Theoretical Optimality.
In the optimal control theory, there are well-known partial differential equations called Hamiltonian-
Jacobi-Bellman (HJB) equations, which assure the theoretical optimality of control agents.
If the control agents can solve the HJB equation, the proposed method attains the optimal state V t ( X αt ) = inf α J t ( X αt ) =
J t ( X α (cid:63) t ) .
However, the optimal agents α (cid:63) of the proposed CSDE with gradient descent are not generally equivalent to the solution to the HJB equation.
In Section 3.4, we propose the Markov Backward conditional (MBcond) loss to assure the optimality of control agents and to provide information in backward dynamics for regularization.
3. M ARKOV F ORWARD C ONDITION
In this section, we ﬁrst raise the important question: Why is the well-posed conditional estimation in cost functional important to accurately train Neural SDE (CSDE) models?
To elucidate the importance of this question, we consider the following minimization problem with the cost functional with naive partial information:
[Equation]
where y ( · ) = { y t } t ∈ [0 ,T ] denotes a set of observed data, and y 0 is the initial data at time t = 0 .
In (6), the conditional expectation is taken to the single initial state X
= y 0 , and the control agents minimize the accumulated losses using this partial information.
As pointed out by Kidger et al.
(2020), this naive cost functional causes a problem when dealing with high-dimensional complex datasets.
This is because the Neural CSDE should disentangle the inherent latent information of complex high-dimensional data to generate accurate results, but the control agents are trained with only the restrictive and partial information of the observed data ( i .
e ., initial condition X 0 = y 0 ).
To solve this problem, we introduce a novel loss function called the MFcond loss that can fully exploit the information of the given observed data y ( · ) , while keeping the Markov structure of X αt :
Deﬁnition 1.
( MFcond loss )
We deﬁne the prediction operator T αs,t as follows, for s < t ,
T αs,t := | I ( s, t ) |
[Equation]
where I ( s, t ) :
= { m : s ≤ t m < t } , | I ( s, t ) | is the cardinality of I ( s, t ) , and { W ( m ) u } m ∈
I ( s,t ) denotes the Wiener processes with respect to time u .
Let us deﬁne a random stopping time
τ s such that τ s :=
inf t { t : l ( t, T αs,t ) > (cid:15) } for the pre-determined threshold (cid:15) 2 .
Then, we can deﬁne the MFcond loss with the stopping time τ ( · ) , as follows:
[Figure]
[Equation]
where χ ( s ) is an indicator function that produces values at the observed time ( i .
e ., χ ( s )
= 1 if y s is observed at s ; otherwise, χ ( s )
= 0 ).
This function is used to consider the irregularly sampled data points.
In (8), naive running cost l of (6) is replaced with l ◦ T αs,τ s , in which the MFcond loss recursively accumulates the expected future losses l ◦ T αs,τ s conditioned on multiple observations.
At each time s , stopping time τ s decides the future time to stop the CSDE propagation by determining if the accumulated losses are larger than the predetermined threshold (cid:15) or not.
While the proposed loss requires a set of multiple conditions on the Markov process X αt to train control agents, information is utilized to generate time-sequential data, and complex dynamics can be expressed.
A conceptual illustration of the proposed MFcond loss is shown in Figure 1-(a).
The main idea of our MFcond loss in (8) is to minimize the differences between the future estimations
X α,su for any given s ≤ u .
In other words, the proposed CSDE is trained to generate an identical future estimation of X αu given any past initial conditions
X α ( · ) =
y ( · ) , i .
e .
, ( X α,su ≈ X α,tu , ∀ s ≤ t ≤ u ) to estimate network inference with multiple conditions in the test time.
This idea is used to introduce a novel inference procedure to overcome the raised issues on the partial information.
Network Inference.
Let { y t m } be the observed data sequences until the current time t in the test dataset.
Our objective is to predict the future points { ˆ y t k } , ( t m ≤ t < t k ).
Our model generates the stochastic estimation ˆ X t k to approximate ˆ y t k at a future time given multiple initial conditions ˆ y t m :
[Equation]
Network Inference
In (9), each control agent makes decisions on its specialized temporal state and collaborates to generate a stochastic conditional estimation ˆ X αt k and approximate ˆ y t k .
As our MFcond loss induces identical estimations X α, ˆ y tm t k for any t m , ˆ X αt k utilizes multiple conditions { ˆ y t m } and fully exploits the past information to predict/estimate future values.
A conceptual illustration of the network inference is shown in Figure 1-(b).
While the proposed inference mechanism utilizes enlarged information 3 compared to a single initial condition, it can model the complex time-sequential data.
If the control agents are trained with the naive cost functional, the terminal states X α,su (conditioned on initial state X s = y s ) and X α,tu (conditioned on initial state X t = y t ) are largely different, which causes problems when we generate complex time-sequential data during the test time, whereas our inference mechanism introduced in (9) utilizes averaged multi-decisions X αt k given different initial conditions.
Thus, the MFcond loss is essential for utilizing the proposed inference procedure.
Unlike the dynamical auto-regressive probabilistic models ( e . g ., ODE-RNNs) that encode whole (or partial) data sequences, as shown in (1), the proposed Markovian CSDE model only uses the current observation to propagate stochastic dynamics.
An additional inference mechanism coordinates the multi-conditioned trajectories to utilize information and produces complex time series.
3. M ARKOV B ACKWARD C ONDITION
In the previous section, we suggested the Markov forward conditional loss that exploits the entire information of time-series data to generate accurate results.
Aside from its empirical beneﬁts to some applications, no theoretical/empirical optimality of (4) is assured by minimizing the MFcond loss in general.
To tackle this problem, in this section, we further introduce the additional stochastic dynamics relating optimality of proposed CSDE-TP.
Let us deﬁne the auxiliary process Z t =
V ( t, X α (cid:63) t ) with a value function V , where α (cid:63) denotes the optimal control agents.
Subsequently, we consider the following forward-backward stochastic differential equations (FBSDEs):
[Equation]
FBSDEs
The ﬁrst SDE ( i .
e .,
X α (cid:63) t ) called the forward SDE has an identical form of (1) and propagates stochastic evaluation in the forward direction with optimal control agents.
The second SDE ( i .
e ., Z t ) called backward SDE recursively subtracts the running cost from the terminal state Ψ
( X α (cid:63) T ) in the backward direction using forward estimations
X α (cid:63) t and cancels the effect of martingales in the diffusion term.
We utilize the property of backward dynamics Z t to train the control agents for the following reasons.
1) Backward Multi-conditions.
Like the MFcond loss with multi-conditions in the forward direction, we want to provide additional information to backward dynamics to train the control agents.
2) Approximated Solution of HJBE.
The auxiliary process Z t gives the theoretical optimality for control agents related to the HJB equation based on the results developed in Yong & Zhou (1999);
Pardoux & Tang (1999), where the process Z t =
V ( t, · ) admits a solution of the HJB equation in (11) and induces an optimal solution for the minimization problem inf α J in (4). [Equation]
[Equation]
where V ( T, x ) =
Ψ( x ) .
In (11), we want to approximate Z t using control agents for optimality.
However, the process Z t requires optimal control agents α (cid:63) that cannot be obtained during the training time.
To overcome this problem, we approximate the auxiliary process Z t with Z αt parameterized by neural control agents α ( · , · , θ ) , which is deﬁned as the modiﬁed version of Z t .
In particular, Z t can be expressed in the following integral form:
[Equation]
where J is the cost functional deﬁned in (3), and ∇ J denotes the gradient of the cost functional with respect to its spatial axis.
Using the proposed process Z αt , we introduce a novel loss function called the MBcond loss to satisfy the two objectives discussed above. [Table] [Table] [Equation]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Equation]
[Equation]
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Table]
[Table] Deﬁnition 2. ( MBcond loss ) Let us deﬁne the auxiliary process Z α t as the solution to (12).
Then, the MBcond loss can be deﬁned as follows:
[Equation]
Theoretically, if we optimize the MBcond loss (13) according to the proposed backward dynamics Z αt , the PDE reformulation of backward dynamics, called Non-linear Feynmann-Kac , have the identical solution 4 to HJB equation in (11).
Thus, our method can attain the optimal solution of original problem posed in section 3.2.
Intuitively saying, one can show that the MBcond loss is equivalent to the reformulation of the minimization problems in (4) using It ˆ o’s formula.
Thus, solving the minimization problem inf α L b induces an identical effect to solve the original problem inf α J .
The only difference is that we utilize multiple conditions to provide conditional information on the backward dynamics Z αt for the regularization of control agents trained with forward conditional dynamics and to impose constraints on control agents, which induces an approximated solution to the HJB equation.
3. O BJECTIVE F UNCTION
In this section, we describe the overall training procedure, which incorporates all the proposed components ( i .
e ., Markov-DP with CSDE-TP, MFcond loss, and MBcond loss) as follows: [Equation]
[Equation]
where L f and L b are deﬁned in (8) and (13), respectively, and γ is a balancing hyperparameter.
In (14), the control agents α =
[ α 1 , · · · , α M ] are trained with a convex combination of MFcond and MBcond losses.
Using the property of CSDE-TP with Markov-DP, the original problem is approximated with the collection of M sub-problems, and each control agent is separately trained with M gradient descent schemes.
Algorithm 1 describes the detailed procedure of our method.
E XPERIMENTS
Network structure of control agents.
The neural network structure for each agent control consists of 2-layers of fully-connected layers, where each module has 128 latent dimensions.
For the activation units, we used the specialized module LipSwish , Chen et al.
Kidger et al.
(2021), to stabilize the FBSDEs during training.
Please refer to Appendix A.6 for detailed information on the network architecture.
Datasets.
For the evaluations, we used PhysioNet, Speech Commands, Beijing Air-Quality, and S&P500 Stock Market datasets.
Refer to Appendix A.5 for data statistics and prepossessing procedures.
Please refer to Appendix A.4 for the discussion on theoretical optimality induced by the MBcond loss.
[Table]
T IME -S EQUENTIAL D ATA R ECONSTRUCTION
[Table]
In this experiment
[Table]
we compared our model against baseline dynamic models: [Latent ODE, Chen et al.
(2018)], [Latent SDE, Li et al.
(2020)], [ODE-RNN, Rubanova et al.
[GRU-D, Che et al.
(2018)], [mTAND, Shukla & Marlin (2021)], and [ODE 2 VAE, C¸a ˘ gatay Yıldız et al.
We used open-source codes provided by the authors for comparison.
For the Latent ODE (SDE) methods, RNN and ODE-RNN were used for the encoder structures, where the decoder structures were identically set to ODE (SDE).
Table 1 shows the performance of all baseline methods compared to the proposed CSDE-TP for the reconstruction tasks.
As evaluation metrics, we used mean squared errors (MSE) and negative log-likelihood (NLL) with open-source code in Rubanova et al.
As shown in Table 1, the proposed method consistently outperformed the baseline methods by a large margin.
In this experiment, we observed that latent dynamics-based methods ( e . g ., Latent ODE/SDE with RNN and ODE-RNN encoders) on models attained similar performances.
We set the latent dimensions of each control agent to 128 for both the reconstruction and prediction experiments.
In the experiments on both datasets, the Mckean-Vlasov (MV) type of the SDE model slightly improved the performance, where it subtracted the mean ( i .
e ., mean-shifting) of the control agent outputs to normalize/reduce the intrinsic volatility in the inferred process ˆ X αt k .
4. T IME -S EQUENTIAL D ATA P REDICTION
[Table]
In this experiment, all the methods were trained with the ﬁrst 36 time-sequences ( e .
g ., 80%) on the training dataset, and evaluated to predict the remaining 12 time sequences ( e .
g ., 20%) on the test dataset.
We followed a procedure similar to that suggested by Rubanova et al.
(2019) to predict the future time series.
Table 2 compares the performance in terms of MSE and NLL, in which the proposed method considerably outperforms the baseline methods.
In the experiment on the Air quality dataset, we set a smaller latent dimension of 64 as the feature dimension compared to those of the other datasets.
Interestingly, the CSDE-TP-MV model exhibited worse performance than the vanilla CSDE-TP model when the data dimension was very small.
This result indicates that the mean-shifting in a low-dimensional data space reduces the expressiveness of the CSDEs.
4. U NCERTAINTY E STIMATION ON S TOCK M ARKET D ATASET
When high volatility is observed over the temporal/spatial axes, conventional evaluation metrics such as MSEs hardly capture the stochastic property of the time-series variations.
Thus, to capture the stochasticity, we evaluated the distance between the distributions of the test data and the inferred/generated data using the maximum mean discrepancy (MMD).
We followed the protocol
[Table]
suggested by Li et al.
(2017) to evaluate the MMD distance, where we used two Gaussian RBF kernels with bandwidths of [5 . 0 , 10 . 0] .
Using this evaluation metric, we experimented on reconstruction tasks using the S&P-500 Stock Market dataset.
Table 3 shows that the proposed CSDE-TP outperforms baselines and effectively recovers the distributional information of stock prices with the stochastic property of the SDE models and the proposed optimization framework.
Interestingly, the latent SDE model attains better performance compared to the Latent ODE, as it utilizes an additional Wiener process to model the data uncertainty.
The performance improvement of the Latent SDE vanishes when we remove the diffusion term ( σ = 0 ) of the latent SDE.
4. E MPIRICAL S TUDY
[Figure]
Efﬁciency of the Markov-DP-TP framework.
To show the empirical advantages of our CSDE-TP model with Markov-DP learning schemes, we evaluated our CSDE-TP according to a different number of control agents on the prediction task using the Air Quality dataset.
Figure 2-(a) shows the training MSEs for several variants of the proposed model in the ﬁrst 20 epochs, where CSDE-TPS hallow1, - S hallow2, and - D eep ( i .
e ., black, blue, and red lines) denote the proposed models with a different number of control agents, i .
e .,
M = 2 , 8 , and 48 , respectively.
The standard CSDE model ( i .
e ., the black dashed line) utilized a single agent, M = 1 .
For all models, the total number of training parameters was equivalently set to ≈ 40 K , and the number of parameters was normalized.
As shown in Figure 2-(a), despite using the same number of parameters, employing multiple agents clearly outperforms the standard CSDE in terms of the learning curve.
From this fact, we can conclude that the Markov-DP-TP signiﬁcantly increased the network efﬁciency compared to the standard CSDE, which indicates that our Markov-DP framework is crucial for training controlled dynamics models.
Efﬁciency of the MFcond loss.
In this experiment, we show the empirical advantages of the multiconditioned CSDE in (8) against the naive partial-conditioned CSDE in (6).
Similar to previous experiments, the results were obtained for the prediction task with the Air Quality dataset.
Figure 2-(b) shows the model conﬁdence in testing MSEs for the ﬁrst 50 epochs, where shaded areas indicate the conﬁdence regions ( i .
e ., ± std ).
The proposed MFcond loss exhibits considerable performance improvement ( . 08 (cid:28) .
compared to the conventional native cost functional and reduces the variances in loss landscapes with stable learning.
With the theoretical discussion in Appendix A.3, we conclude that the proposed CSDE actively exploits the information of the complex time series with multiple conditions to accurately generate complex time-series.
C ONCLUSION
In this paper, we introduce a novel Markov-type CSDE with the TP function that records the individual attention of each control agent at sub-intervals along the temporal axis.
Using the properties of the
CSDE and TP, we suggest Markov DP to efﬁciently train the control agents by decomposing the original problem into smaller sub-problems.
To overcome the practical/theoretical issues, we propose two novel losses, namely, MFcond and MBcond losses.
The MFcond loss captures the future time to estimate the running costs, while multiple conditions are actively provided to forward dynamics.
The MBcond loss assures the theoretical optimality of the control agents and imposes regularization by providing additional information to backward dynamics.
Experimental results demonstrate the efﬁciency of the proposed method for various tasks
A A PPENDIX A. D ETAILED C OMPARISON TO E XISTING M ETHODS
In this section, we investigate the relation between our method and existing methods.
Reverse SDE vs. Backward SDE.
Song et al.
(2020) suggested a novel SDE called reverse SDE , which shares semantically similar idea with BSDE: both reverse/backward SDEs enhance the forward
SDE by providing additional information to drift/diffusion functions in forward dynamics.
The mathematical motivation of the reverse SDE in Anderson (1982) is to pose the SDEs with Wiener processes W t , ˆ W t with respect to these minimal increasing/decreasing sigma algebras A t , ˆ A t and deﬁne the relation between them:
[Equation]
where X t is a solution to the forward SDE and p t is the probability density of X t .
Using the relation in (15), the reverse SDE transforms the prior distribution ( e .
g ., Gaussian noise distribution)
back into the data distribution ( e . g ., 2D images) by gradually removing the noises and reconstruct the original data with the well-designed score function ( i .
e ., ∇ p t ( x ) ) in backward dynamics.
In contrast to the reverse SDE, the role of backward SDE in this paper is to consider the probabilistic reformulation to access the cost functional to provide the additional information in backward dynamics.
Stacked ODE vs. CSDE-TP.
Massaroli et al.
(2020) suggested the stacked Neural ODE that shares similar idea with the proposed CSDE-TP, where temporally piece-wise neural nets are considered to model the complex dynamics.
However, the stacked ODE faces the aforementioned problem on partial conditional information when generating complex data as their models only take initial values to propagate dynamics.
As opposed to their models, the proposed model is trained with multiple observations and directly generates time series in data space without any latent embedding network.
Furthermore, we generalize their optimal control problems to the stochastic version and propose the
Markov-DP-TP framework that can systemically solve the problem.
DDPMs vs. CSDE-TP.
Tashiro et al.
(2021) suggested denoising denoising diffusion models (DDPMs) that are conditioned on the set of the observed data, where the generated sequential data is assumed to be gradually transformed from an initial state in the forward direction and the backward process is parameterized by the neural network and trained to minimize speciﬁc ELBOs.
Speciﬁcally, the transition probability p θ ( X t − 1 | X t ) in the backward process is deﬁned as a parameterized Gaussian distribution:
[Equation]
(16) where mean and covariance ( µ θ , σ θ ) are parameterized by the neural network θ .
Similar to the proposed CSDE, their parameterized functions are closed-loop type processes and the whole probabilistic sequential model p θ is posed as the Markov chain: p θ ( X 0: T ) =
p θ ( X T ) (cid:81) T t =1 p θ ( X t − 1 | X t ) .
In contrast to the DDPM, the probability transition in the proposed CSDE is deﬁned as the continuous generalization called controlled Fokker-Planck equation (CFPE):
[Equation]
(17) where t > s ∈ [0 , T ) and x, y ∈ R d , p t ∼ X αt is the probability distribution of X αt .
The CFPE in (17) one-to-one corresponds to the CSDE ( i .
e ., X αt ).
Compared to the discrete-time Gaussian transition model, this conditional probability can express complex continuous-time probability transitions while maintaining the Markov structure.
A. N OTATIONS AND B ACKGROUND
We ﬁrst deﬁne the basic deﬁnitions of probabilistic objects:
Deﬁnition 3. A ﬁltration {F t } is an increasing sequence of {F t } of σ -algebra such that F 0 · · · , ⊂ F t ⊂ F . The triplet (Ω , F t , P ) is called a ﬁltered probability space.
Deﬁnition 4. The ﬁltration generated by Wiener process W t is deﬁned as F Wt = σ { W 0 , · · · , W t } . In this case, W t is naturally F Wt -adapted by construction. Deﬁnition 5.
The stochastic process { X t } is called {F t } -adapted if X t is F t measurable for every 0 ≤ t ≤ T .
Throughout this paper, we work on the ﬁltered probability space (Ω , {F t } t ∈ [0 ,T ] , P ) with the ddimensional F t -Wiener process W t and natural ﬁltration F Wt .
We assume that α i for all 1 ≤ i ≤ M is admissible Markov control , ( i .
e ., α i is F t -adapted and α i ∈
A i , X αt has a unique solution).
Deﬁnition 6.
(Markov Process)
Let X t be a F t -adapted stochastic process.
Then, X t is the Markov process if the following equality holds:
[Equation]
Deﬁnition 7.
(Controlled Stochastic Differential Equation)
[Equation]
The solution to the above CSDE is denoted as X α,st .
If the initial states are speciﬁed ( i .
e ., starting point X s = x ), we denote the solution as X α,s,xt .
By the deﬁnition of Markovian control agents, in all cases, the solution to the proposed CSDE in (1) is a Markov process.
Mathematical Assumptions.
In this paper, we assume that functions b, σ are uniformly Lipschitz continuous along its spatial axis and
bounded (cid:107) b ( t, 0; · ) (cid:107) , (cid:107) σ ( t, 0; · ) (cid:107) at the entire interval [0 , T ] .
We assume that each functions b i ( · , x, · ) , σ i ( · , x, · ) , Ψ( x ) , l ( · , x ) are twice differentiable for all 1 ≤ i ≤ M , ( i .
e ., , b i , σ i , Ψ , l ∈ C 2 ( R n ) .
and both drift and diffusion functions are uniformly Lipschitz on its spatial axis, i . e . b i , ∂ x b i , ∂ 2 x b i , σ i , ∂ x σ i , ∂ 2 x σ i ∈ Lip ) , and the trainable parameters of the control agents α i , θ i are lying in the compact subset C of its ambient space ( i .
e ., θ i ∈ C ⊂ R m ).
A. E NLARGED I NFORMATION BY C OLLECTION OF O BSERVED D ATA
In the proposed inference procedure, we deﬁne a novel operator T in (9) to consider the multiconditioned dynamics with the Markov-type SDE model.
Although this operator plays a central role in the paper, its mathematical properties have not been carefully dealt and investigated thoroughly.
In this section, we discuss the relation between this operator and the enlarged information that is obtained by collecting past observations.
In addition, we generalize the inference mechanism in (9) to a mathematically rigorous form and discuss the effect of the proposed operator T by showing some probability inequality.
Suppose that we have two observed conditional states { X t m } , { X t n } until the current time t , ( t n , t m < t < t k ) and the objective is to predict/generate the future value
y t k using this information.
We consider the deterministic time t k
by replacing random stopping time τ t m to simplify the discussion.
First, we deﬁne the two-parameter stochastic process Y to model the proposed operator
T in an alternative way:
[Equation]
[Equation]
where w ∈ Ω takes a value in the probability space.
The stochastic process Y is the ( F t m ∨ F t n ) - valued random variable for any ﬁxed t m , t n < t by the deﬁnition, where F t m ∨F t [Equation] is the composited smallest sigma algebra with two ﬁltrations.
In the deﬁnition, we assume that processes X α,t m 1 ,t k , X α,t n 2 ,t k are derived from two independent Wiener processes W t and ˆ W t .
Then, we can deﬁne the two-parameter martingale Zakai (1981); Khoshnevisan (2003) in the following form:
[Equation]
By the deﬁnition of M , it can be easily shown that M is the reformulation of the MFcond loss for some ﬁxed number of past observations.
Note that M is truly a martingale because conditional estimations are summed in the deﬁnition of T .
The control agents are trained to minimize the M given the information induced by past observations ( i .
e ., composited ﬁltration { (cid:87) t m <t F M tm } ), which indicates that the proposed inference procedure can infer the future value ˆ X αt k according to the enlarged information { (cid:87) t m <t F M tm } .
By the fact that M is a martingale with respect to the composited ﬁltration, we obtain the following result using Doob’s maximal inequality:
[Equation]
where the inequality shows that errors between the future value y t k and the generated samples X α,tt k at time t
k are bounded by the maximal perturbation probability.
As the control agents are trained to minimize the MFcond loss ( i .
e ., M ) in the right-hand side of inequality, it renders the probabilistic bound of L 2 errors at future time t k .
A. D ETAILED D ISCUSSIONS ON THE MB COND LOSS
In this section, we investigate the detailed theoretical structure of the MBcond loss and its fundamental rationale for the optimality of control agents.
For this, we rephrase the cost functional in the general form:
[Equation]
The classical non-linear Feynman-Kac theorem in Yong & Zhou (1999) states that given the cost functional J with the control agents α , one can obtain the second-order parabolic partial differential equation from (23): [Equation]
[Equation]
where (cid:104)· , ·(cid:105) denotes the inner product and the boundary condition is given as J ( T, x )
=
Ψ( x ) .
Subsequently, by applying It ˆ o’s formula to (23), we obtain the following probabilistic formulation:
[Equation]
By rearranging each term above
[Equation]
the backward stochastic differential equation is induced directly.
[Equation]
Note that, in the main paper [Equation] we use the inverse sign convention, (cid:82) T t ( · ) = − (cid:82) t T ( · ) , to emphasize the backward direction.
By using the formulations [Equation] the MBcond loss in (13) can be rewritten in full description as follows:
[Equation]
where y ( · ) =
( y 1 , · · · , y T ) ∼ p
( y 1 , · · · , y T ) denotes the set of observed data.
The regularization effect comes from the expectation evaluation of the third term in (26).
Speciﬁcally, one can obtain the following equality by using the Itˆo’s isometry:
[Equation]
Because the MBcond loss is posed to minimize this additional martingale term in (28) in backward dynamics according to the forward dynamics X αt , it reduces the over-conﬁdence of generated timeseries.
By the relation ( t, x ) → J ( t, x ) → Z αt → L b ( t, x ) for any ( t, x ) ∈ [0 , T ] × R + , the update rule for the MBcond loss can be expressed as follows:
[Equation]
where this formulation is similar to (5) and shows that gradient descent with respect to θ r for the
MBcond loss can be explicitly deﬁned.
Admissible Control Set A .
In previous discussions, we show the relation between J with BSDE dynamics Z t and the well-deﬁned gradient descent.
The next step is to deﬁne the proper control set A to relate the gradient descent with optimality.
Let us deﬁne the Hilbert space L 2 (cid:44) { ϕ ( t, x ; ¯ θ ); R n -valued F t -progressively measurable ∀ ¯ θ ∈ C } with the norm (cid:107) ϕ (cid:107) 2 L 2 = E (cid:104)(cid:82) T 0 | ϕ
( t, x ; θ ) | 2 dt (cid:105) < ∞ .
We assume that control agents α r is L αr
Lipschitz on the parameter variable, i .
e ., (cid:13)(cid:13)(cid:13) α r ( · , · ; θ rk, 1 ) − α r ( · , · ; θ rk, 2 ) (cid:13)(cid:13)(cid:13)
L 2 ≤ L αr (cid:13)(cid:13)(cid:13) θ rk, 1 − θ rk, 2 (cid:13)(cid:13)(cid:13) for any θ rk, 1 (cid:54)
= θ rk, 2 ∈ R m and any 1 ≤ k ≤ K .
In all case, we assume that any θ rk lies in the compact subset C of R m .
Each functions b i ( · , x, · ) , σ i ( · , x, · ) , Ψ( x ) , l ( · , x ) are twice differentiable for all 1 ≤ i ≤ M , ( i . e ., , b i , σ i , Ψ , l ∈ C 2 ( R n ) . and both drift and diffusion functions are uniformly Lipschitz on its spatial axis, i .
e .
b i , ∂ x b i , ∂ 2 x b i , σ i , ∂ x σ i , ∂ 2 x σ i ∈ Lip ) . As we deﬁned Ψ and l as usual Euclidean distance, regularity/uniform lipschitzness for these functions are trivial. For the ﬁxed parameter θ , we deﬁne the r -th control agent as θ r → α r ( · , · , θ r ) (cid:44) α r ( θ ) ∈
L 2 .
Truly, the image space of α r ( θ ) is the closed subspace of the Hilbert space L 2 due to the Lipschitzness with compactness of θ .
Let θ r ( k ) :
N → R m be the trajectory for the training parameters of the r -th control agent at learning iteration k .
Without loss of generality, J r [ α r ] = J ( t, X ( α r ,α ( − r ) ) t ) .
We deﬁne the Euclidean closed metric balls { B δ kr } k ∈ N centered at θ ( k ) with the radius δ kr < ∞ such that B δ kr = { ϑ ∈
R m ; (cid:107) ϑ − θ r ( k ) (cid:107) ≤ δ kr , θ r ( k ) is local minimum of J r [ θ ( k )] } .
Let us consider the sub-sequence { θ (¯ k ) } ¯ k ∈ ¯ N ⊆
θ r ( k ) } k ∈ N , which induces the strictly-decreasing cost functional { J r [ θ (¯ k )] } ¯ k ∈ ¯ N with the ordered index set ¯ N .
Then, the admissible control set A is deﬁned as follows:
[Equation]
where ¯ K is the maximal element in ¯ N and the constant K ∈ N indicates the last iteration index of training deﬁned in Algorithm 1.
Intuitively, the control set A can be understood as a collection of local minimum obtained by M gradient descent schemes during training.
[Equation]
31) where V ∈ C 1 , 2 ([0 , T ] , R d ) . By the deﬁnition of metric balls { B δ k } and strictly-decreasing properties, the inﬁmum in (31) is attained when θ ( K ) =
θ and the control agent α
( θ ( K )) is optimal in this control set.
Relation to Stochastic Maximum Principle (SMP).
We consider an arbitrary control in the convex set K ∈ A with β ∈ K and the optimal control α ( θ ( K )) .
Let DJ | β = dd(cid:15)
J ( θ ( K ) + (cid:15) ( β − α )) | (cid:15)
= be the G ˆ ateaux derivative (this can be deﬁned while the control set is a vector sub-space, A ⊂ L 2 ).
By the result of Pontryagin maximum principle, Theorem (4.12) in Carmona (2016a), one can obtain the inequality as follows:
[Equation]
for t ∈ [0 , T ] almost surely, where we deﬁne H (cid:44) H ( t, X t , Y t , Z t , α t ) for the Hamiltonian system H with adjoint variables Y t , Z t and deﬁne the arbitrary control β
= α ( · , · , θ ) ∈ A with some θ .
The ﬁrst inequality is satisﬁed due to the deﬁnition of Lipschitzian control agents.
The optimality condition indicates converging upper-bound of DJ | β to 0 .
In our method, the optimality condition of the proposed learning framework is bounded by the
Euclidean distance between θ ( K ) and θ in parameter space.
Thus, the proposed framework poses a fundamentally different approach to interact with optimality conditions in SMP.
As we deﬁne that the θ ( K ) is a local minimum of J with the inequality (cid:107) θ ( K ) − θ (cid:107) <
δ Kθ , the gradient descent scheme that induces the tight radius { δ kθ } k ∈ N assures optimality by the relation 0 ≤ DJ | β ≈ δ Kθ .
Relation to the HJB equation.
We consider the inﬁnitesimal generator L t of the non-homogeneous controlled Markov process X t as L αt f =
(cid:104)∇ f, b ( t, x, α ) (cid:105)
+ 12 Tr (cid:2) σσ T ( t, x, α ) ∇ 2 f (cid:3) .
We show the important relation between the proposed MBcond loss with the HJB equation as follows:
[Equation]
Equivalence Relation
In the left-hand side of (33), the PDE formula is directly consequence of the non-linear Feynman-Kac theorem that we derive in (24).
The distinct point is that control agents are obtained by the gradient descent of the MBcond loss with BSDE ( i .
e ., Z t ).
Note that, as shown in (31), θ ( K ) is actually an optimal control.
This means that, without heavy calculations to solve the PDEs, the gradient descent algorithm also assures optimality of control agents in the proposed control set A .
In contrast, the HJB equation in the right-hand side states that the optimal control agent can be obtained by solving the second-order parabolic formula and the inﬁmum is taken by considering algebraic properties of candidates for the exact solution.
If the solution to HJBE exists in the control set A , the PDE in the left hand side of (33) approximates the solution to the HJB.
Overall, we argue that the MFBcond loss can provide a novel deep learning-based paradigm to adopt/solve the conventional stochastic optimal control problem in a feasible way ( i .
e ., well-deﬁned loss functions with the gradient descent scheme).
A. D ATA P REPOSSESSING
PhysioNet dataset , Silva et al.
(2012), contains overall 8000 multivariate time series obtained for the ﬁrst 48 hours of a different patient’s admission to intensive care unit (ICU).
Each patient has a set of 35 various clinical features.
We normalized features of all patients in the dataset to have zero mean and unit variance.
We used a half of time-series as the training dataset and the remaining parts as the test dataset.
Speech Commands dataset , Warden (2018), consists of one-second audio records of various spoken words such as “Yes”, “No”, “Up”, and “Down”.
Since there were nearly 100,000 record samples, we sub-sampled the dataset due to the dimensionality of training instances on two conﬂicting classes ( i .
e ., “Right” and “Left”).
Overall, 6950 time-series records were selected, where 80% were used as training dataset and the remaining parts as the test dataset.
We pre-processed these time series by computing Mel-frequency cepstrum coefﬁcients from the audio signal, so that each time series was spaced with 65 and 54 channels.
Then, we normalized each channel of all signals in the dataset to have zero mean and unit variance.
Beijing Air-Quality dataset , Zhang et al.
(2017), consists of multi-year recordings of air quality data across different locations in Beijing.
Each sample contains 6-dimensional time series features of
PM 2 . 5 , PM 10 , SO 2 , NO 2 , CO, and O 3 , which are recorded per hour.
We segmented data to have the length of 48 and normalized each feature of all data in the dataset to have zero mean and unit variance.
S&P-500 Stock Market dataset consists of stock market data with 6 -dimensional feature vectors ( i .
e ., [High, Low, Open, Close, Volume, Adj Close]).
For the complete data acquisitions, we excluded enterprises with incomplete recordings during sampling duration, thus total 381 enterprises are selected.
The time-series are sampled every 30 -min with T = 48 temporal states.
Similar to Speech commands dataset, we used ﬁrst 80% of temporal states to train the model and the remaining parts are used for prediction task.
A. E XPERIMENTS D ETAILS
Different SDE candidates for CSDE. Owing to the abstract form of the proposed CSDE in (1), various types of drift and diffusion functions ( i . e ., b and σ ) can be selected according to different applications. In Table 4, we enumerate candidate functions.
In the experiments, we adopted two models: Vanilla and Mckean-Vlasov (MV) SDEs.
Hyperparameters.
For the running and terminal costs ( l and Ψ , respectively), we used the ldistance, i .
e ., l ( s, x ) = (cid:107) x − y s (cid:107) 22 and Ψ( x ) = (cid:107) x − y T (cid:107) 2 .
In all experiments, γ is set to 0 .
To estimate the gradient of the MBcond loss, we estimated numerical gradients with the auto-grad library in Pytorch (Paszke et al.
[Table] Network Architecture for Neural Control Agents. Each control agent α i ( t, X t ; θ i ) has an identical neural network architecture, which consists of linear layers and non-linear units.
Figure 3 shows the detailed network architecture.
Each agent takes concatenation of temporal/spatial tensors ( t, X t ) as its input, where the temporal tensor t is transformed into new form t (cid:48) by the time inhomogeneous embedding layer.
We followed the setting suggested in Park et al.
(2021) for this embedding.
After time embedding, concatenated tensor ( t (cid:48) , X t ) is fed into two Linear layers with nonlinearity units ( i .
e .,
LipSwish in Chen et al.
Kidger et al.
Finally, the transformed tensors are split into the control terms for drift and diffusion functions.
The diffusion functions are deﬁned as non-degenerate types, where σ i ( t, X t , α i ) =
Diag ( z t ) and z t is the output of the last linear layer.
The latent dimension of each Linear layer was set to 128 in all experiments except for the prediction task with the Air Quality dataset ( = 64 ).
Thus, a total number of training parameters for single control agent α
i is ≈ 11 K .
Simulation of CSDE and Temporal Privacy Function.
Let T = t ∈ { t k } 1 ≤ k ≤ N for the pre-ﬁxed time interval ∆ t .
We apply the Euler-Maruyama scheme to approximately simulate the proposed
CSDE:
[Equation]
where Z (cid:44) Z (0 , √ ∆ t I d ) is a d -dimensional Gaussian random variable with zero mean and covariance √ ∆ t I d .
Analysis of Instability at Contact Points.
In every time stamps t , drift and diffusion functions are controlled over neural control agents α i where we assume that t − and t + are adjacent points of contact point t with inﬁnitesimally small duration. The process A t indicates drift integral term and σ αs denotes the diffusion term in our forward CSDE dynamics. As shown in the inequality, the Markovian property is still preserved, and the magnitude of jumps are controlled by Lipschitzness of drift/diffusion functions.
[Equation]
In a probabilistic point of view, the set of contact points may be regarded as measure-zero, and the probabilistic evaluation is not changed.
Figure 4 shows a particular example, where 14 temporal states ( i .
e ., | T | = 14) with 4 control agents are considered.
In the ﬁgure, the black line indicates the trajectory of time series, blue dots denote the observed data points, and shaded grey dots denote the missing data points.
Each control agent takes 5 data points, where 2 temporal states are shared to other agents.
In the experiments, the total number of temporal privacy functions are maximally set to M = | T | / 2 , where each control agent shares 2 points for smooth transitions of stochastic dynamics.
A. A DDITIONAL E MPIRICAL S TUDY
[Figure]
Effect of hyper-parameter γ .
In the Figure 5-(a), the effect of hyper-parameter γ is shown.
Similar to Fig 2-(b), the results were obtained for the prediction task with the Air Quality dataset.
Each red, black, blue line indicates the test MSE for different γ ∈ [0 . 0 , 0 .
95 , 1 . 0] over 50 epochs.
If the
MFcond is deactivated during the training time i .
e ., γ = 0 .
0 , only MBcond loss is utilized to train the proposed CSDE-TP, and the model produces poor results.
As our inference procedure requires the model to train with multiple conditions, the obtained result seems obvious.
If the MBcond loss is deactivated during the training time i .
e .,
γ = 1 . 0 , multi-conditioned information in backward dynamics Z αt are canceled, and the performance is decreased signiﬁcantly i .
e ., 1 .
277 → 2 . 003 .
This clearly shows that MBcond loss boost the performance.
Effect of random stopping time.
In the Figure 5-(b), the effect of strategy to select (cid:15) is shown.
If we select threshold (cid:15) as an uniform random variable (cid:15) ∼ U [ s, T ] which is independent to X t , then the network quickly falls into instability as shown red line in Figure 5-(b).
This shows that the well-designed strategy for selecting threshold (cid:15) is crucial factor to stabilize the network learning landscape.
Contrary to the random sampling strategy, our method deﬁned in Algorithm 1 select half value of maximal MFcond loss in last learning steps as the threshold (cid:15) for random stopping time ( i .
e ., 12 max l k − 1 → (cid:15) k ).
As the threshold (cid:15) is always bounded above the maximal loss in the last steps, random stopping time at iteration k is decided in the time set:
[Equation]
36) where τ ks denotes the stopping time at learning iteration k .
If the network trains the MFcond loss so that l k (cid:44) l ( t, T αs,tk ) → 0 as training proceeds k → ∞ , then it is clear that the stopping time vanishes τ k →∞ s ∈ ∅ .
Thus, the strategy 36 is well-deﬁned.
A. D ETAILED E XPLANATIONS OF M ARKOV DYNAMIC PROGRAMMING WITH TEMPORAL PRIVACY
For the clear explanation of proposed Markov-DP-TP, let us consider the detailed example.
we decompose sub-problem ( B (cid:48) ) in equation (4) into another smaller sub-problems:
[Equation]
[Equation]
where we set α ( − r )
=
β , w r ( s ) = 1 t ≤ s ≤ u .
In this case, the problem ( B ) on interval [ u, T ] is now decomposed into smaller sub-problems ( C ) , ( C (cid:48) ) on two intervals [ u, u (cid:48) ] and [ u (cid:48) , T ] .
Similarly to u in the equation (4), another auxiliary time index u (cid:48) is considered here for additional problem ( C ) .
The corresponding new temporal privacy function w r (cid:48)
( s ) =
1 u ≤ s ≤ u (cid:48) is deﬁned on the interval [ u, u (cid:48) ] .
By repeating temporal decomposition of original problem ( A ) M times, one can ﬁnd the following hierarchical relations:
• P1 ) .
original problem, T =
[ t, · · · , T ] , α , no temporal privacy • P2 ) .
Two sub-problems ( B ) + ( B (cid:48) ) , equation (4), Time set, T = [ t, · · · , u, · · · , T ] , control agents, α =
[ α r , α ( − r ) ] , temporal privacy functions = { w r } • P3 ) .
Three sub-problems ( B ) + ( C ) + ( C (cid:48) ) , Time set, T =
[ t, · · · , u, · · · , u (cid:48) , · · · , T ] , control agents, α =
[ α r , β, β ( − r (cid:48) ) ] , temporal privacy functions = { w r , w r (cid:48) } • P4 ) .
M sub-problems, ( A ) + ( B ) + ( C ) + . . .
, Time set, T =
[ t, · · · , T − tM , · · · , r ∗ T − tM , · · · , T ] , α
=
[ α 1 , α 2 , · · · , α r , · · · , α M ] , temporal privacy functions = { w 1 , w 2 , · · · , w r , · · · , w M }
The role of u in equation (3) and (4) is replaced to u and u (cid:48) in ( P3 ) , and replaced to ( r ∗ T − tM ) in ( P4 ) in the table if the time interval is assumed to be regularly sampled.
Similarly, the role of r in the equation (3) and (4) is replaced to r and r (cid:48) in ( P3 ) .
A. T OY E XAMPLE ON S YNTHETIC D ATA
In this section, we conduct the reconstruction experiment on synthetic data to show the different behaviors and demonstrate the advantages of the proposed CSDE compared to previous methods.
Stochastic Trigonometric Data.
In this experiment, we deﬁne the 100 -dimensional stochastic process with composition of trigonometric functions ( i .
e ., sin , cos ) as follows:
[Equation]
where we assume t ∈ [0 , 1 . 0] and the total number of temporal states are set to 48 ( i .
e ., | T | = 48 ).
In the deﬁnition of synthetic process Y t , both the period and amplitude are randomized with mean-zero
Gaussian random variables ( i .
e ., Z 1 ∼ N (0 , 1 . 0) , Z 2 ∼ N (0 , 2 . 0) , Z 3 ∼ N (0 , 12 I d ) ).
With the effect of Gaussian random variables, the process contains high volatility in both the spatial/temporal axes.
We compare our method to the auto-regressive ODE-RNN Rubanova et al.
(2019) model using the open-source code implemented by the authors.
To observe the fundamental difference between
ODE-RNN and CSDE-TP, we stop the training procedure when the estimated MSEs of both models attained the threshold ( ≤ . 07) .
In Figures 6 and 7, the ﬁrst axis of trigonometric data are visualized.
The results of each model are indicated as the blue lines ( i .
e ., X t ) and the synthetic trigonometric data is indicated as the red lines ( i .
e .
, Y t ).
The 95% -conﬁdence regions ( i .
e ., CR-95) of both the test and predicted time-series are shown as red and blue shaded regions, respectively.
ODE-RNN.
Figure 6 shows the results of the ODE-RNN model.
Although the ODE-RNN model attains relatively similar MSEs compared to the proposed model, there are two main issues in their model to be discussed.
1) It hardly captures the vertical perturbation of test data induced by Z 3 and the obtained result produces a small variance at every temporal states.
2) It hardly captures the horizontal perturbation of test data induced by Z 1 , Z 2 , and the obtained result produces the temporally unmatched trajectories
These phenomenons occurred due to the deterministic property of the ODE-RNN model, where the dynamical transition in their model is posed as the ODEs that cannot express the stochastic variation.
CSDE-TP.
Figure 7 shows the result of the proposed CSDE-TP model and shows the advantages of adopting the SDE in modelling stochastic dynamics.
Compared to the results of the ODE-RNN, the proposed method accurately captures both the vertical/horizontal perturbations and recover the 95% conﬁdence region.
It is clear that our CSDE-TP delicately expresses the complex volatility of stochastic trajectories.
Discussions.
As aforementioned in Section 4.3, experimental results on synthetic stochastic data show that the MSE is not the best metric to train/evaluate the time-series models if there exists the high volatility in the dataset.
In this case, distributional metrics such as MMD and Wasserstein distance can be good substitutes for training/evaluating stochastic data.
A. F UTURE W ORK
We plan to extend the proposed CSDE model to a general controlled Markov Itˆo-L´evy jump diffusion model (Øksendal & Sulem (2007)) to delicately express the complex time-sequential data.
For example, the proposed CSDE can be generalized to the Markov Itˆo-L´evy jump diffusion of the following form: [Equation]
where N ( t, z ) = (cid:80)
[Equation]
s ≤ [Equation] X z ∈ U ( η s − η s − ) , and Poisson random measure η t .
As the previous work in Jia & Benson (2019) show the effectiveness of the jump process in modelling complex discontinuous dynamics, we believe this generalization will produce the comparable results and broaden our understanding in modelling dynamical systems for time-series data.