Learning Continuous-Time Dynamics by Stochastic Differential Networks
Authors: Yingru Liu, Yucheng Xing, Xuewen Yang, Xin Wang, Jing Shi, Di Jin, Zhaoyue Chen
Keywords: Continuous-time Stochastic RNN, Neural SDE
Abstract: Learning  continuous-time  stochastic  dynamics  is  a  fundamental  and  essential problem in modeling sporadic time series, whose observations are irregular and sparse in both time and dimension.  For a given system whose latent states and observed data are high-dimensional, it is generally impossible to derive a precise continuous-time  stochastic  process  to  describe  the  system  behaviors.  To  solve the above problem, we apply Variational Bayesian method and propose a flexible continuous-time stochastic recurrent neural network named Variational Stochastic Differential Networks (VSDN), which embed the complicated dynamics of thesporadic time series by neural Stochastic Differential Equations (SDE). VSDNs capture the stochastic dependency among latent states and observations by deep neural networks.  We also incorporate two differential Evidence Lower Bounds to efficiently  train  the  models. Through  comprehensive  experiments,  we  show  that VSDNs  outperform  state-of-the-art  continuous-time  deep  learning  models  and achieve remarkable performance on prediction and interpolation tasks for sporadic data.