Paper: Feynman-Kac Operator Expectation Estimator: An Innovative Method for Enhancing MCMC Efficiency and Reducing Variance (ICLR-2024)

Reviewer comment (Presentation):
The paper's presentation is poor, lacking the algorithm and numerical results in the main text. It makes the proposed method inaccessible to the readers.

Author response:
A1 The detailed algorithm and numerical results are indeed provided in the appendix.

Reviewer comment (Writing):
Some notation is used in Algorithm 1 without proper definition, e.g., MSE(.,0).

Author response:
A2 We have addressed this issue in the revised version.

Reviewer comment (Writing):
How is MSE(., 0) is defined?

Author response:
A2 We have addressed this issue in the revised version.

Reviewer comment (Theory):
Add more explanations or theoretical proofs for the statement that ``the solution to the Feynman-Kac equation at the initial time is E(f(X_T)|X_0=x_0).

Author response:
A3 We have addressed this issue in the revised version. This is a fundamental conclusion, and a concrete proof can be found at [1]  
[1] Applied stochastic differential equations. S Särkkä, A Solin - 2019 - Cambridge University Press .page 118

Reviewer comment (Writing):
The current state of writing makes it very hard to grasp the gist of the proposed methodology.

Author response:
Thank you very much for your response. We sincerely apologize for not being aware initially that resubmission of the paper was possible, and thus, we did not submit a revised version. We welcome any suggestions on how to improve the quality of writing and the overall flow of the paper. We will make the necessary revisions and submit an improved version shortly.

Reviewer comment (Presentation):
In Section 3.1, a key aspect of the methodology seems to solve a bridge-matching problem with respect to the samples of some MCMC chains. However, the specifics of the authors’ approach are missing from the main text. Only after looking at Algorithm 1 in the appendix does it become clear that some loss functions are being minimized. The definition of these loss functions should have been in the main text.

Author response:
We appreciate the feedback and acknowledge the need for greater clarity regarding the specifics of our approach in solving the bridge-matching problem. We agree that the definition of the loss functions, which play a crucial role in our methodology, should be presented in the main text for better understanding. We will revise the manuscript accordingly to provide a more detailed explanation of these key components in Section 3.1. Thank you for bringing this to our attention.

Reviewer comment (Theory):
In Section 3.2, the authors do not provide enough explanation, motivation, or proof as to why the reverse solution of the Feynman-Kac equation is the right conditional expectation. The paper is expected to be self-contained to a certain degree.

Author response:
We acknowledge the importance of making the paper self-contained to a reasonable extent. To address this concern, we will provide additional details, motivation, and, if necessary, a concise proof in Section 3.2. Additionally, we will include a reference to the relevant proof, particularly in [1] p118, and consider adding this result to the appendix for the readers' convenience. Thank you for highlighting this, and we will ensure that the necessary information is appropriately presented in the revised manuscript.

Reviewer comment (Theory):
The paper claims that the proposed methodology "necessitates fewer assumption since it does not rely on the law of large numbers and the Markov ergodic theorem." However, it does rely on a strong assumption: that the PDE can be solved. The paper does not contain any proof of the accuracy of the solution of the proposed differential equation. Even if the differential equation's solution is the desired conditional expectation mathematically, discretizing and numerically solving it is a completely different challenge. This is crucial in an MCMC setting, since empirically assessing the quality of samples and expectation estimates is very challenging.

Author response:
Firstly, In stochastic analysis, if an SDE is well-defined in the sense that it has a strong solution, then the corresponding probabilistic PDE (FK equation) must have a unique solution.The well-definedness of SDEs is a fundamental condition for any SDE-based sampler (diffusion bridge model). For example in Langevin MCMC the Lipschitz property for the potential energy function is a fundamental assumption. Our approach does not enhance any of the conditions. And it weakens the $P$ and $f$ assumptions. Secondly, regarding the error estimation of the PDE solver, we will provide references to relevant literature, particularly in [2], where appropriate error estimates for PDE solvers are discussed.  
Finally , We can observe the effectiveness of our approach in the isingmodel example. Despite the additional computational requirements for training the SDE parameters and solving the PDE, the overall time remains within a reasonable and controllable range, especially when leveraging techniques such as multi-threading and parallelization. In contrast, using only the MCMC expectation estimator would require well over 10^7 points in the Markov chain for n=5 in https://github.com/zysophia/Doubly_Adaptive_MCMC/blob/main/data/isingcompare_complexity.csv . For larger values of $n$, utilizing the MCMC estimator becomes practically infeasible.  

[1] Applied stochastic differential equations. S Särkkä, A Solin - 2019 - Cambridge University Press  

[2] Tim De Ryck and Siddhartha Mishra. Error analysis for physics-informed neural networks (pinns) approximating kolmogorov pdes. Advances in Computational Mathematics, 48(6):79, 2022

Reviewer comment (Theory):
One of the key motivations seems to be replacing the need for discarding burn-in samples. However, it is unclear how this is true when the paper tries to match the output of an MCMC estimator. This output will contain non-asymptotic bias that can only be resolved through burn-in. Therefore, trying to match this output may end up inheriting the bias.

Author response:
Firstly, we can observe from the solution of the PDE that only the points at the initial time correspond to the terminal expectation. For training the PDE, we need to use points from the burn-in period, and the matching error can be controlled through optimization of the loss function (for cases with analytical density, this error is zero). And matching errors are present in any diffusion model, including diffusion model, Score-based SDE,Flow match ODE. There are numerous theoretical results analyzing the matching error, and it is unrelated to bias.  

Secondly, how do we reduce bias? Our approach differs significantly in how we utilize the sampled points. Classical MCMC estimators use information from the points themselves, denoted as $X_t$, while we leverage the gradient information $(\mu(t,X_t),\sigma(t,X_t))$. This error is induced by the training of the Physics-Informed Neural Network (PINN) and can be controlled through the loss function. In contrast, the bias in classical MCMC estimators arises due to the inherent uncertainty in the sampled points $X_t." For more details, refer to the experiments in Appendix A4 to better understand this point.

Reviewer comment (Novelty):
The idea of connecting MCMC expectation estimators with partial differential equations is not new, with an existing and growing literature [1-4]. While the approaches of these works are different and the reviewer is not claiming that this work lacks novelty in that regard, these works already achieve what the paper seems to attempt: post-processing the output of an MCMC algorithm to improve performance of the expectation estimator. Furthermore, these methods come with asymptotic and non-asymptotic consistency guarantees and work well in practice. The paper should have compared and discussed these methods.

Author response:
Firstly, the utilization of partial differential equations to approximate MCMC expectations represents a classic inverse approach. However, prior methods predominantly followed a forward direction, employing MCMC to solve PDEs. Our proposal introduces a novel perspective by adopting an inverse approach.  

Secondly, existing post-processing methods for MCMC fail to challenge a fundamental assumption, namely the efficiency limitation inherent in MCMC due to the curse of dimensionality, where the optimal variance scaling is limited to $O(n^{1/2})$. In terms of generality, control variables (functions) are often designed for specific problems and are not generic. For example, in the Hamiltonian function in the Ising model, you would be hard pressed to determine an appropriate control function to reduce the variance. Our method breaks free from this assumption through operator approximation, expanding the scope of MCMC applicability across a broader range of probability density functions $P$ and functions $f$. This in itself constitutes a novel and substantial contribution.  

One question - indeed, we reviewed the references you provided, but these are not suitable for computing expectations in random graphs such as the Ising model. We apologize if there was any misunderstanding. The reasons are as follows: Firstly, the Stein methods mentioned in [1,2,3,4] require the density function $\pi $ to be continuously valued, which is not applicable to random graphs. In the Ising model, particles have only two states {+1, -1}, and the joint distribution is inherently discrete. Moreover, there is no differentiability condition such as $\nabla_x \log \pi_t < \infty$, and the normalization constant cannot be determined.  

The second and more crucial issue is that the values of the Hamiltonian function are also discrete. However, it is practically impossible to determine all possible values of the Hamiltonian function since there are too many potential values, a total of $2^{n^2}$ cases. If $n$ is sufficiently large, it becomes infeasible to ascertain the entire function's value set. In such cases, it is challenging to optimize and determine a control function through optimization methods.  

[1]Oates, Chris J., Mark Girolami, and Nicolas Chopin. "Control functionals for Monte Carlo integration." Journal of the Royal Statistical Society Series B: Statistical Methodology 79.3 (2017): 695-718.  

[2]Sun, Zhuo, Alessandro Barp, and François-Xavier Briol. "Vector-valued control variates." International Conference on Machine Learning. PMLR, 2023.  

[3]South, Leah F., et al. "Semi-exact control functionals from Sard’s method." Biometrika 109.2 (2022): 351-367.  

[4]South, Leah F., et al. "Regularized zero-variance control variates." Bayesian Analysis 18.3 (2023): 865-888.

Reviewer comment (Experiments):
The claimed advantage of the proposed method seems to be doutable, or at least not supported in this paper. It is said that standard MCMC approaches suffer from long burn-in period, but the proposed approach requires training of neural network models at both the diffusion bridge model stage and the PDE solving stage, which to me seems a much higher cost.

Author response:
We can observe the effectiveness of our approach in the isingmodel example. Despite the additional computational requirements for training the SDE parameters and solving the PDE, the overall time remains within a reasonable and controllable range, especially when leveraging techniques such as multi-threading and parallelization. In contrast, using only the MCMC expectation estimator would require well over 10000000 points in the Markov chain for n=6 in https://github.com/zysophia/Doubly_Adaptive_MCMC/blob/main/data/isingcompare_complexity.csv . For larger values of $n$, utilizing the MCMC estimator becomes practically infeasible.

Reviewer comment (Theory):
The claim of reduced variance from the proposed approach seems not supported, and the approximations due to imperfect training of neural networks, discretization of PDE, etc, would cause a systematic bias towards the computed expectation. How much the bias is, and whether it can be bounded are also not discussed.

Author response:
This is a theoretical concern, and our paper proposes a viable method validated through classical examples presented in Appendices A1 and A4. Experimental results demonstrate a reduction in bias, as evidenced in A4. The efficiency of our approach is proven in A1. Regarding the estimation of errors due to the imperfect training of neural networks and PDE discretization, relevant theoretical discussions are available in [1] and other associated literature. It is essential to acknowledge that our paper primarily introduces a practical method and highlights its empirical success, and the theoretical aspects are supported by existing literature and experimental results.

Reviewer comment (Evaluation):
Most MCMC samplers are targeted for generating samples from the distribution, which could be used for posterior inference etc. The proposed approach, however, can only be used to compute the expectation of one function. Say we want to compute the posterior mean, posterior variance and a 95\% posterior credible interval, then the proposed approach have to be used repeatedly for each quantity, while standard MCMC approaches only require one sampling chain.

Author response:
This is a valuable and insightful point. Indeed, our proposed approach focuses on computing the expectation of a specific function. For different statistical quantities, such as posterior mean, posterior variance, or a 95% posterior credible interval, we would need to use our approach repeatedly, requiring retraining when changing the function $f$ in the boundary conditions. While the training for different ffs is independent, utilizing parallel computing can efficiently address this issue. An even more radical solution is to explore mapping PDE solutions and boundary conditions directly, for example, through methods like DeepOnet. This could potentially overcome the need for retraining when changing the function $f$

Reviewer comment (Experiments):
Even if we don't want to use standard MCMC samplers that involves a burn-in period, we have the alternatives of training a normalizing flow / diffusion model / stochastic localization process to sample from the target distribution and then computing the expectation based on the samples. How does the proposed approach compare to these methods, and what are their connections?

Author response:
Firstly diffusion models such as (DDPM,stable diffusion) etc. already exist. The models all require a burn-in period.The specific reason for this is because it is impossible to draw unconditional distributions for SDE sampling, and any terminal distribution is actually a conditional distribution $X_T$ under condition $X_0$. Thus we often need a large $T$ to avoid initial value dependence. This is reflected in the diffusion model by the fact that we need to add very many steps of noise in the forward process to transform the picture into a Gaussian distribution. And in the backward process we iterate more steps to get the generated samples.

Secondly, assuming you are using a stochastic localisation process or normalizing flow, sampling is often easy and fast. But a central question is how you can use the samples to accurately estimate expectations. What kind of expectation estimator is unbiased and minimises variance? How can we estimate with as few sample points as possible? In the traditional MCMC expectation estimator, there are only two estimators without considering control variables, and we have already discussed the problems with these two estimators. Also under such models such as normalizing flow, there is often no ergodicity theorem, which means that only that one estimator based on the law of large numbers can be used. Our biggest contribution is to propose a generalised expectation estimator, this one can handle the diffusion bridge model and can maximise the use of these picked points to help us estimate the expectation better. This expectation estimator requires far fewer points than the traditional MCMC expectation estimator and weakens the $P$ and $f$

This can be seen using the observation in terms of information encoding, firstly we are given a $P$ or some points, the diffusion bridge (diffusion model) or MCMC algorithm, is to encode the information in $P$ into $\mu,\sigma$. And we calculate the expectation $E[f(X_T)]$ actually trying to decode the statistics from $\mu,\sigma$. Direct sample averaging is just a method of decoding, but directly using the gradient information from $\mu,\sigma$ is a more efficient way of decoding.

Reviewer comment (Writing):
The text is much too informal and needs to be more precise; having the pseudo code only in the appendix worsens this problem.

Author response:
Pseudo Code Placement: We understand the concern regarding the placement of pseudo code in the appendix. We will reconsider its placement in the main body of the manuscript for better accessibility and understanding.

Informality and Precision: We acknowledge the need for increased precision in the language used throughout the manuscript. We will revise the text to ensure a more formal and precise tone, making it accessible to a wider audience.

Reviewer comment (Presentation):
In Section 2, the paper cites a number of works in which diffusions are employed in disparate contexts, and summarising these works as discussing the same "diffusion model" seems strange.

Author response:
The citation of works in Section 2 aims to categorize various models under the umbrella term "diffusion bridge model." This generalization is motivated by the shared characteristic of sampling from a target distribution, whether its density is known or unknown. The commonality lies in their potential for efficiency improvement using FKEE.

Reviewer comment (Writing):
Section 3.1 is unclear; it is difficult to distinguish which methods are actually used to find the parameters of the diffusion process and which methods are merely "alternative" possibilities not actually implemented by the authors.

Author response:
Response to Concerns in Section 3.1:
Target Distribution Classification: We have classified the target distribution into two scenarios: known density and unknown density. This categorization serves as the foundation for discussing various diffusion bridge models applicable to each case.

Examples of Diffusion Bridge Models: We have provided examples of diffusion bridge models for both scenarios. These examples illustrate the diversity of models available for parameter estimation in different contexts.

Introduction of Neural SDE Method: We propose a method based on Neural Stochastic Differential Equations (SDE), fundamentally a diffusion bridge model. Unlike other methods, this approach stands out as it can handle target distributions with both known and unknown densities, and it can reconstruct the underlying Markov chain. Further advantages are detailed after the statement, "We consider using it for the following reasons".
We emphasize that all mentioned diffusion bridge models, including the Neural SDE method, have the potential to use FKEE for estimating expectations, replacing traditional direct averaging methods. Experimental validations have been conducted for some methods, affirming the applicability of FKEE.

Reviewer comment (Evaluation):
The contributions list on Page 2 makes strong claims, such as that the proposed methodology "can estimate expectations without succumbing to the limitations imposed by the curse of dimensionality", "often exhibits superior efficiency" (without specifying compared to what), and "leads to a more efficient utilization of Markov chains" while necessitating fewer assumptions since it does not rely on the law of large numbers and the Markov ergodic theorem. These claims are not convincingly substantiated in the manuscript.

Author response:
"Can estimate expectations without succumbing to the limitations imposed by the curse of dimensionality":
Refers to the efficiency of PDE solvers in estimating expectations, particularly in high-dimensional spaces, avoiding the challenges associated with the curse of dimensionality.

"Often exhibits superior efficiency" (compared to what?):
Compared to classical Markov Chain Monte Carlo (MCMC) expectation estimators.

"Leads to a more efficient utilization of Markov chains. Notably, it necessitates fewer assumptions since it does not rely on the law of large numbers and the Markov ergodic theorem":
Utilizes gradient information of $\mu(t,X_t)$ and $\sigma(t,X_t)$ at the corresponding position of $X_t$, allowing for more efficient use of Markov chains with fewer reliance on assumptions compared to traditional MCMC methods based on the law of large numbers and the Markov ergodic theorem.

These observations are validated through experiments with the Ising model. In the high-dimensional case, such as when $d=255$ (already highly dimensional for a graphical model), where the partition function involves summing 2^255 discrete points, our method remains applicable. The enhanced efficiency of our method, FKEE, is evident as it utilizes significantly fewer points on the Markov chain compared to the classical MCMC expectation estimator. This improvement is particularly notable in the comparison of sample points for $wi$, sample points for $vi$​, and the MCMC estimator in [1]. For further verification, one can refer to the [1] method under corresponding parameter configurations, noting the number of points on the Markov chain needed https://github.com/zysophia/Doubly_Adaptive_MCMC. In this model setup, our method is the only one that reaches $n=15$ and uses only 2000 points on the Markov chain with lower error and less time. In contrast, the current state-of-the-art expectation estimator for MCMC only reaches the $n=5$ case. This is already a big improvement. Meanwhile, our method can be trained using GPU acceleration, which is an advantage not available to the classical MCMC expectation estimators, which can only get the estimation by means of more point picking and control functions.

[1] hahrzad Haddadan, Yue Zhuang, Cyrus Cousins, and Eli Upfal. Fast doubly-adaptive mcmc to esti-
mate the gibbs partition function with weak mixing time bounds. Advances in Neural Information
Processing Systems, 34:25760–25772, 2021

Reviewer comment (Theory):
The proposed methodology likely introduces approximation errors at numerous stages because the constructed diffusion only approximately admits \(P\) as stationary distribution, the sample paths generated from the diffusion suffer from discretisation error, and the solution of the Feynman–Kac formula is only approximated via deep learning.

Author response:
Q1 the constructed diffusion only approximately admits as stationary distribution

A1 "The constructed diffusion only approximately accepts $P$ as a static distribution" is a fundamental challenge inherent in all Markov chain Monte Carlo (MCMC) methods and one of the reasons for the existence of burnout periods. But you may note Table 1: Comparison of Diffusion bridge models. not all of them require $P$ as a static distribution ($T=\infty$), and in these models ($T<\infty$) only the expectation estimator of the Law of Large Numbers can be used because of the lack of the conditions of the ergodicity theorem. Our approach can handle these.

Q2 sample paths generated from the diffusion (which are needed for approximately finding the solution of the Feynman--Kac formula) suffer from discretisation error; the solution of the Feynman--Kac formula is only approximated via deep learning.

A2  The acknowledgment that "sample paths generated from the diffusion (needed for approximately finding the solution of the Feynman--Kac formula) suffer from discretization error" is indeed a fundamental concern. This issue constitutes a central focus in contemporary diffusion models, such as DDPM, Score-based SDE, and other diffusion models. A substantial body of theoretical work has addressed the bounds of this discretization error, particularly in scenarios where only a discrete set of observations is available. It is worth noting that in cases where the density is known, as in our method with accurate $\mu$ and $\sigma$, this issue is non-existent. A classic example illustrating this point is Langevin MCMC The solution of the Feynman-Kac formula is approximated using deep learning, and theoretical estimates of the associated errors can be found in [1], as referenced in our related work.

Reviewer comment (Experiments):
The proposed method appears to require running at least one MCMC chain long enough to obtain some samples from the target distribution \(P\). This raises the concern that the problem might already be solved at that stage; unless the authors can demonstrate that their approach, despite substantial additional computations needed for training SDE parameters and solving the PDE, outperforms simply running the MCMC chain longer, its advantage is unclear.

Author response:
Q4 It seems to me that the proposed method requires to run at least one MCMC chain long enough to obtain some samples from the target distribution . 

Whether or not to run a chain depends on the type of diffusion bridge model, when the density has an analytical formal expression we do not need to match the diffusion bridge model, just run a small number of chains (5 tracks were used in the experiment in A4) to get the sample $X_t$ and then compute $\mu(t,X_t),\sigma(t,X_t)$. We only need to match diffusion bridges if we don't know that the density exists only at discrete points, which is what diffusion models currently do (DDPM, normalised flow method), but this method takes longer to sample, and generating more samples (using the law of large numbers method to estimate the expectation) will take even longer. Our approach improves the efficiency of this method.

We haven't fully addressed the problem simply by obtaining a sufficient number of samples through running an MCMC chain. The key challenge lies in how to precisely compute mathematical expectations using these samples, a task influenced by various factors mentioned in the Discussion section of our paper (4 discussion). Classical MCMC estimators, which are currently available, are limited. To correct bias and reduce variance in these estimators, a large number of samples is typically required. This constitutes the core issue we investigate. Our method specifically focuses on maximizing the utilization of a limited number of MCMC samples to estimate expectations effectively.

However, can the authors demonstrate that their approach (which incurs substantial additional computations needed for training the SDE parameters and solving the PDE) outperforms running the MCMC chain for just a while longer?
We can observe the effectiveness of our approach in the isingmodel example. Despite the additional computational requirements for training the SDE parameters and solving the PDE, the overall time remains within a reasonable and controllable range, especially when leveraging techniques such as multi-threading and parallelization. In contrast, using only the MCMC expectation estimator would require well over 10000000 points in the Markov chain for n=6 in https://github.com/zysophia/Doubly_Adaptive_MCMC/blob/main/data/isingcompare_complexity.csv . For larger values of $n$, utilizing the MCMC estimator becomes practically infeasible. In such cases, would you still consider the MCMC expectation estimator to be efficient?


This can be seen using the observation in terms of information encoding, firstly we are given a $P$ or some points, the diffusion bridge (diffusion model) or MCMC algorithm, is to encode the information in $P$ into $\mu,\sigma$. And we calculate the expectation $E[f(X_T)]$ actually trying to decode the statistics from $\mu,\sigma$. Direct sample averaging is just a method of decoding, but directly using the gradient information from $\mu,\sigma$ is a more efficient way of decoding.