Paper: Variational Inference for SDEs Driven by Fractional Noise (ICLR-2024)

Reviewer comment (Novelty):
Perhaps variational inference may not be the main contribution of the paper since it is still a result of Girsanov's theorem like presented in Li et al. 2020. However, I believe that the idea of finding SDE approximation outweights this point.

Author response:
We thank the reviewer for their helpful comments and appreciate that they find our introduction to be excellent. We agree with the reviewer that our variational inference resembles earlier works such as Li et al. (2020) and is a direct application of Girsanov's theorem. However, to benefit from this strong, established result, we did need to show how a new SDE driven by BM can be obtained from an SDE driven by fBM. As the reviewer confirms, this is where the novelty lies (in addition to our novel scheme of weight optimization). Moreover, our variational inference framework paves the way to our video prediction application. We have incorporated all the minor suggestions.

Reviewer comment (Theory):
Can you elaborate on the difference between a Type I and a Type II fBM and which form delivers benefits in which circumstances?

Author response:
$\textbf{Difference between a Type I and a Type II fBM.}$ 
Type I, also called Mandelbrot-Van Ness or 'standard' fBM is the most well known definition of fBM.
Type II, also called Riemann-Liouville fBM, is historically most prevalent in econometric literature.
As can be seen in the integral definitions (Eq. (3) and (4)), Type II omits the first integral from $-\infty$ to $0$ in the definition of Type I.
So Type II is, in a sense, a simplification. Yet, to the best of our knowledge, its covariance has no simple analytical expression. On the other hand, the Type I covariance is the straightforward, well known Eq. (1).

As described by [5,6] the main difference is that Type I has stationary increments, and Type II has non-stationary increments. This means that Type II has a larger emphasis on the origin $t=0$, which might not be favourable for some applications. For example, if during training we sample a sequence from a video dataset at a random start point, this $t=0$ has no special or distinguished meaning and should not be treated differently by the driving fBM process. In other words, Type I ensures a shift in time has no effect on its increments. However, this is not the case for Type II.
This difference is relevant for our framework, since the increments are driving the SDE.
We thank the reviewer for bringing this up and have added our explanations to the appendix.

Reviewer comment (Theory):
It would be of importance to comment on the assumptions that guarantee a finite KL term inside of the ELBO, i.e., the diffusion terms of prior and posterior have to be equal.

Author response:
$\textbf{A finite KL term inside of the ELBO and Novikov's conditions.}$
It is seen in Equations (13) and (16) that the diffusion terms of prior and posterior have to be equal. This holds by construction. However, to derive an explicit finite KL term in our ELBO, we are required to invoke the Girsanov theorem, which necessitates a second technical condition, namely the Novikov condition. 
The reviewer is right that the control term $u(\cdot)$ learned via a neural network is assumed to satisfy Novikov's condition. 
In fact Chen et al. [7] combined with the appendix in https://openreview.net/attachment?id=wG12xUSqrI&name=supplementary_material (Lemma 8) indicates two sufficient conditions for Novikov to hold:
* sub-Gaussian tails for the initial samples ($X_0$ or $Z_0$), which is easy to enforce
* uniform Lipschitz continuity of the neural network

The second is challenging for us to guarantee. This is similar to earlier works performing stochastic variational inference, which either use an approximation argument [7] or assumes that Novikov’s condition holds at the outset [8,9,10]. In our work, we admit the latter.

Reviewer comment (Theory):
According to (Oksendal 2003, The Girsanov Theorem II), the control function $u$ needs to meet Novikov's condition. In the proposed work $u$ is parameterized by a neural net. How is the aforementioned condition controlled during training?

Author response:
$\textbf{A finite KL term inside of the ELBO and Novikov's conditions.}$
It is seen in Equations (13) and (16) that the diffusion terms of prior and posterior have to be equal. This holds by construction. However, to derive an explicit finite KL term in our ELBO, we are required to invoke the Girsanov theorem, which necessitates a second technical condition, namely the Novikov condition. 
The reviewer is right that the control term $u(\cdot)$ learned via a neural network is assumed to satisfy Novikov's condition. 
In fact Chen et al. [7] combined with the appendix in https://openreview.net/attachment?id=wG12xUSqrI&name=supplementary_material (Lemma 8) indicates two sufficient conditions for Novikov to hold:
* sub-Gaussian tails for the initial samples ($X_0$ or $Z_0$), which is easy to enforce
* uniform Lipschitz continuity of the neural network

The second is challenging for us to guarantee. This is similar to earlier works performing stochastic variational inference, which either use an approximation argument [7] or assumes that Novikov’s condition holds at the outset [8,9,10]. In our work, we admit the latter.

Reviewer comment (Writing):
Can you elaborate more on the following: "Unlike Tong et al. (2022) our approach is agnostic to discretization and the choice of the solver."

Author response:
$\textbf{On discretization and the choice of the solver.}$
We would like to clarify the distinction to Tong et al. (2022), which mainly differs from our work in two ways: (i) fractional Brownian motion (fBM) is approximated as a a Gaussian process (GP), (ii) only the Type II representation of fBM is used as as an integral over increments of the Wiener process. Tong et al. performs a finite time discretization of the Type II integral to obtain a first approximation of the increments of FBM. In a second step, this approximate GP is further approximated using a sparse GP approach based on a smaller set of pseudo or inducing points which are distributed over time. Conditioned on the inducing points, samples from the sparse GP are independent random variables at each discrete time point. Finally, this (conditioned) white noise process is further interpreted in terms of the Euler discretisation of an ordinary SDE leading to effective drift and diffusions. For the latter SDE, one can apply Girsanov's theorem and the corresponding ELBO (conditioned on the inducing points) to perform inference. 

Note, that their current derivation of effective drift and diffusion relies on the Euler discretisation of SDE. For higher order SDE solvers, the approximation has to be adapted, which requires new derivations. As a main difference, in our paper, the approximation is not based on the discretisation in the time domain but of the discretisation of an integral representation (our equation (5)) over a spectrum of decay constants of Ornstein-Uhlenbeck (OU) processes (driven by the same Wiener noise). Since each OU process already represents a noise process with temporal correlations, we can expect that a linear combination of a small number of such processes can yield a good approximation of the covariance of fBM over some given time interval. Our approximation leads to a system of SDEs (without conditioning) for which the ELBO can be easily obtained. Since the time discretisation of the resulting SDE is performed $\textit{after}$ the OU approximation, any SDE solver can be directly applied.
With this flexibility, in our paper, we have chosen the second order Stratonovich–Milstein solver.

Reviewer comment (Experiments):
The experiment lacked sufficient data, and the basis for asserting the presence of long-term dependency was inadequate.

Author response:
$\textbf{On experimentation.}$
We thank the reviewer for bringing this up. We have now conducted an additional evaluation on the $\textbf{real video dataset of chaotic double pendulum}$, as suggested by the Reviewer J9Kf. Note that this dataset comprises of real video captures of a double pendulum in motion. We provide the details in our first response to J9Kf as well as in our $\textbf{revised Sec. 5 and Appendix}$. Overall, results on this dataset further reveal the merits of fractional noise in capturing the nuisances of real world.

Note that this new dataset and SM-MNIST contain different forms of nuisances and present different challenges to our stochastic model. First, SM-MNIST digits move with a constant velocity along a trajectory until they hit at wall at which point they bounce off with a random speed and direction. This sudden event intersperses the deterministic motion with moments of uncertainty, i.e. each time a digit hits a wall. This is the reason why a stochastic model fits better than an ODE and unlike BM, our noise can model the smooth and correlated trajectory simply by raising the Hurst index.

On the other hand, the double-pendulum dataset is actually governed by a set of coupled ordinary differential equations. However, despite being a simple physical system, it exhibits a rich dynamic behavior with a strong sensitivity to initial conditions and noises in the environment (motion of the air in the room, sound vibrations, vibration of the table due to coupling with the pendulum etc.). Combined with the chaotic nature of the system, this creates a major challenge for any model based upon smooth ODEs. Our model on the other hand heavy lifts this difficulty onto the (fractional) stochastic noise, leading to a more appropriate model. As shown (see our new Table in Sec. 5), our model outperforms the BM baseline also in this dataset.

Reviewer comment (Theory):
Why is it believed that Brownian motion cannot capture long-term dependencies, while fBM can?

Author response:
$\textbf{BM vs fBM, Markovian-ness and long range dependencies.}$
Brownian motion is a stochastic process with stationary, independent and normally-distributed increments. Due to this independence, the probability of each event depends only on the state attained and therefore correlations to further away past or future states are not considered, \ie it is a Markov (memory-less) process, $\textit{what happens next depends only on the state of affairs now}$. This can be understood as the noise process (infintesimal increments) being $\delta$-correlated for BM.
The increments of fBm (with Hurst index $H$), on the other hand, are not independent unless $H=1/2$ for which we recover BM. If $H<1/2$ then the increments of the process are negatively correlated. If $H>1/2$ then the increments of the process are positively correlated, shows $\textit{long-range dependence}$ (LRD), usually related to the $\textit{rate of decay in statistical dependence}$ of two points with increasing time interval. For fBM, the rate of decay follows a power law, indicating that the process is non-Markovian. Please see [1] for a simple yet rigorous proof of this. 

In general, we agree that one may have long time dependencies for ordinary stochastic differential equations (without fractional noise). However, for larger classes of such models, e.g. the so called reversible diffusions, one obtains an exponentially fast relaxation of fluctuations under certain asymptotic conditions on potentials (well explained in the book by Pavliotis [2]).
On the other hand, even simple linear systems driven by fBM, such as the the fOU process, inherit a non exponential, power law decay of stationary correlation functions from the driving process.

Reviewer comment (Theory):
Why does fBM encompass non-Markovian dynamics?

Author response:
$\textbf{BM vs fBM, Markovian-ness and long range dependencies.}$
Brownian motion is a stochastic process with stationary, independent and normally-distributed increments. Due to this independence, the probability of each event depends only on the state attained and therefore correlations to further away past or future states are not considered, \ie it is a Markov (memory-less) process, $\textit{what happens next depends only on the state of affairs now}$. This can be understood as the noise process (infintesimal increments) being $\delta$-correlated for BM.
The increments of fBm (with Hurst index $H$), on the other hand, are not independent unless $H=1/2$ for which we recover BM. If $H<1/2$ then the increments of the process are negatively correlated. If $H>1/2$ then the increments of the process are positively correlated, shows $\textit{long-range dependence}$ (LRD), usually related to the $\textit{rate of decay in statistical dependence}$ of two points with increasing time interval. For fBM, the rate of decay follows a power law, indicating that the process is non-Markovian. Please see [1] for a simple yet rigorous proof of this. 

In general, we agree that one may have long time dependencies for ordinary stochastic differential equations (without fractional noise). However, for larger classes of such models, e.g. the so called reversible diffusions, one obtains an exponentially fast relaxation of fluctuations under certain asymptotic conditions on potentials (well explained in the book by Pavliotis [2]).
On the other hand, even simple linear systems driven by fBM, such as the the fOU process, inherit a non exponential, power law decay of stationary correlation functions from the driving process.

Reviewer comment (Experiments):
Why was latent video generation chosen for experimentation, and why were experiments not conducted on datasets other than MNIST?

Author response:
$\textbf{Why latent video generation?}$
The main contribution of our work is a new differential equation, driven by the approximate fractional Brownian motion as well as a variational inference framework, whose drift, diffusion and control functions can be learned. 
Due to the stochastic nature of the real world, video perception becomes an appealing and important problem for showcasing the benefits of our method.
Note that, we do not claim that our video architecture, despite being novel, brings an ultimate solution. Yet, it enables variational learning under our neural SDE, which is essential to validate our contributions. In principle, our stochastic differential equations can be applicable to various sorts of stochastic video perception methods.
We thank the reviewer for mentioning this and leave further applications to a future study.

Reviewer comment (Experiments):
The empirical evaluation is of limited scope. Qualitatively, the method is evaluated on 2 toy problems (fOU & Hurst index); quantitatively on a single synthetic dataset (stochastic moving MNIST). For the latter, only two baseline models from 2018 and 2020 are compared to. In consequence, the usefulness of the method is not established. On the plus side, there are 3 ablations / further studies.

Author response:
$\textbf{On empirical evaluations.}$
Following the reviewer's suggestion, we added extra experiments on a $\textbf{real-world dataset composed of videos of the chaotic motion of a two-link double pendulum}$, released by IBM under https://developer.ibm.com/exchanges/data/all/double-pendulum-chaotic/ [1]. We thank the reviewer for this to-the-point suggestion. We provide the results below as well as in our revised Sec. 5:

| Model |   ELBO  | PSNR  |
|-------|:-------:|-------|
| BM    | −545.13 | 26.11 |
| fBM   | −636.61 | 27.09 |

Even though this system is governed by ODEs, its chaotic nature poses a big challenge for any ODE-based method to faithfully capture its behaviour. Arguing that the hard-to-predict transitions can be modeled by stochastic noise, we evaluate our video model using both BM and fBM, where the latter attains a lower ELBO as a higher PSNR. We have $\textbf{revised Sec. 5}$ and incorporated qualitative results indicating the diversity of our samples. We hope that this new result clarifies some of the concerns and strengthens our empirical evaluation.

Reviewer comment (Novelty):
A detailed comparison with Tong et al. (2022) (who also learn approximations to fBM) is missing. So far, it is only stated that Tong et al. did not apply their model to video data and that it is completely different. A clear illustration of the conceptual differences and a comparison of the pros and cons of each approach would be appreciated (summary in main part + mathematical details in supplementary material).

Author response:
$\textbf{Comparisons to Tong et al. (2022).}$ Note that, Tong et al. (2022) only reports Hurst index estimation in their experiments. Following the same protocol, we have already compared our method against theirs in time-dependent Hurst index. This experiment ($\textbf{Fig. 3}$) not only extends our method to $\textit{ multifractional Brownian Motion}$, but also shows that our approach yields more correct Hurst estimates, especially in the extremes. This is further supplemented by our conceptual discussion in $\textbf{App. E.2}$. An important difference is that our work covers Type I, whereas Tong et al. (2022) does not.

Reviewer comment (Reproducibility):
After reading the method and experiment section multiple times, the implementation details are still unclear. Although source code is provided, the paper remains insufficiently detailed in the following regards:  
- The encoding step returns \(h\), a sequence of vectors/matrices over time, which is used to compute \(\omega\), but the relation to the optimal \(\omega\) from Proposition 5 is unclear.  
- \(h\) is passed to a temporal convolution layer producing \(g\), which has as many 'frames' as the input and is used as input to the control function \(u\).  
- Control, drift, and diffusion functions are implemented as neural networks.  
- An SDE solution is numerically approximated with a Stratonovich–Milstein solver.  
- The role and use of \(\omega\) in the decoding step are unclear.  
- The role of the approximation processes \(Y\), their parametrization, whether \(\gamma\) is as in Proposition 5, and whether they are integrated separately from \(X\) are unspecified.  
- The number of sample paths used to estimate the mean in the ELBO expectation is unspecified.  
- In Figure 6, the reason for prior samples always showing a 4 and a 7 is unclear, as is whether the prior depends on the observations.

Author response:
The static content vector $w$, in the explanation of the video modelling framework, should not be confused with the weights $\omega$ (Greek letter omega) in the fBM approximation. The approximation processes $Y_k$ are integrated jointly with $X$. The state of the SDE process is thus constructed as in Eq. (14): $Z = (X, Y_1, \dots, Y_K)$. The values of $\gamma$ are described in our Appendix for all experiments. For the video models, we only take one sample path to calculate the ELBO. For the bridge and time dependent Hurst experiments we take $16$ or $32$ samples.

Reviewer comment (Experiments):
The task and evaluation protocol for the stochastic moving MNIST experiment is not clearly specified. It appears, but is not confirmed, that the model is given a sequence of 25 frames and tasked with returning the same 25 frames, aiming to learn a generative model.

Author response:
$\textbf{On the prior and Moving MNIST.}$ In our video modelling experiments, both the prior model and posterior model are parameterized. The prior is as defined in Eq. (13). The posterior model is the prior model with the added control function $u(t)$, as defined in Eq. (16). Since only the control function receives information of the data sequence, the posterior is capable of reconstructing the sequence during training. When the model is trained, we use the prior as a generative model. Specifically, due to maximizing the ELBO, the prior will learn to model the dataset. For example, to condition on the first frames of a movie and generate predictions, we first use the posterior model on the given frames. At the time point where we start predicting, we use the prior. Practically, this simply means setting $u(t)=0$.

The Peak Signal-to-Noise Ratio (PSNR) metric is calculated frame-wise, averaged over time, by sampling $100$ videos conditioned on the first $5$ frames. The best PSNR is selected from these $100$ samples.
This approach is adapted from SLRVP to allow direct comparison.
This metric is representing how good the model captures the data.
Please note that we have considerably improved the manuscript under the 'Latent video models' section and incorporated this explanation.

Reviewer comment (Experiments):
The method is not evaluated on real world data.

Author response:
$\textbf{On empirical evaluations.}$
Following the reviewer's suggestion, we added extra experiments on a $\textbf{real-world dataset composed of videos of the chaotic motion of a two-link double pendulum}$, released by IBM under https://developer.ibm.com/exchanges/data/all/double-pendulum-chaotic/ [1]. We thank the reviewer for this to-the-point suggestion. We provide the results below as well as in our revised Sec. 5:

| Model |   ELBO  | PSNR  |
|-------|:-------:|-------|
| BM    | −545.13 | 26.11 |
| fBM   | −636.61 | 27.09 |

Even though this system is governed by ODEs, its chaotic nature poses a big challenge for any ODE-based method to faithfully capture its behaviour. Arguing that the hard-to-predict transitions can be modeled by stochastic noise, we evaluate our video model using both BM and fBM, where the latter attains a lower ELBO as a higher PSNR. We have $\textbf{revised Sec. 5}$ and incorporated qualitative results indicating the diversity of our samples. We hope that this new result clarifies some of the concerns and strengthens our empirical evaluation.

Reviewer comment (Experiments):
Quantitatively, the method is only evaluated on one synthetic dataset (stochastic moving MNIST).

Author response:
$\textbf{On empirical evaluations.}$
Following the reviewer's suggestion, we added extra experiments on a $\textbf{real-world dataset composed of videos of the chaotic motion of a two-link double pendulum}$, released by IBM under https://developer.ibm.com/exchanges/data/all/double-pendulum-chaotic/ [1]. We thank the reviewer for this to-the-point suggestion. We provide the results below as well as in our revised Sec. 5:

| Model |   ELBO  | PSNR  |
|-------|:-------:|-------|
| BM    | −545.13 | 26.11 |
| fBM   | −636.61 | 27.09 |

Even though this system is governed by ODEs, its chaotic nature poses a big challenge for any ODE-based method to faithfully capture its behaviour. Arguing that the hard-to-predict transitions can be modeled by stochastic noise, we evaluate our video model using both BM and fBM, where the latter attains a lower ELBO as a higher PSNR. We have $\textbf{revised Sec. 5}$ and incorporated qualitative results indicating the diversity of our samples. We hope that this new result clarifies some of the concerns and strengthens our empirical evaluation.

Reviewer comment (Experiments):
The experimental evaluation would be stronger with more extensive experiments, including evaluation on more tasks and datasets, particularly ones where correlated noise is present (e.g., the pendulum dataset from Becker et al., ICML 2019).

Author response:
$\textbf{On empirical evaluations.}$
Following the reviewer's suggestion, we added extra experiments on a $\textbf{real-world dataset composed of videos of the chaotic motion of a two-link double pendulum}$, released by IBM under https://developer.ibm.com/exchanges/data/all/double-pendulum-chaotic/ [1]. We thank the reviewer for this to-the-point suggestion. We provide the results below as well as in our revised Sec. 5:

| Model |   ELBO  | PSNR  |
|-------|:-------:|-------|
| BM    | −545.13 | 26.11 |
| fBM   | −636.61 | 27.09 |

Even though this system is governed by ODEs, its chaotic nature poses a big challenge for any ODE-based method to faithfully capture its behaviour. Arguing that the hard-to-predict transitions can be modeled by stochastic noise, we evaluate our video model using both BM and fBM, where the latter attains a lower ELBO as a higher PSNR. We have $\textbf{revised Sec. 5}$ and incorporated qualitative results indicating the diversity of our samples. We hope that this new result clarifies some of the concerns and strengthens our empirical evaluation.