Paper: Reverse Diffusion Monte Carlo (ICLR-2024)

Reviewer comment (Experiments):
Evaluations in 2d simply do not meet the bar for a conference paper. Methodologies such as SMC among many other modern ML variants are able to do quite well in multimodal 2d examples.

Author response:
> There is a "diffusion models for sampling" aspect to your algorithm, and whilst prior work is mostly (not entirely e.g. [1]) focused on VI approaches to estimate the score, the overal should be discussed and prior relevant formulations of the score should be credited appropiately.


A key aspect of our work is the exploration of log-Sobolev constant dependency of the gradient complexity, where we have found that rdMC outperform conventional methods. This distinction makes a direct comparison with parameterized models infeasible. We have added discussion in Related work and Appendix A4.


> This is true but notice that prior work [2] establishes quite detailed bounds on the error that these parametric estimators of the score can potentially achieve, and it may be interesting (yet not necessary) to discuss at a high level how the complexity of these bounds compares to the results on the inner loop you have provided.

It is interesting to see the parametric error analysis of diffusion-based sampling algorithms. From a high-level perspective, [2] focus on the expressiveness of neural networks to learn the score function and we focus on the feasiblity and complexity to learn the score function for different time step $t$. Given our algorithm and analysis, if the neural network is well-specified (contains the real score function), the score learning of neural networks can be reformulated as a a statistical problem, where samples are obtained with computation efforts as we mentioned.




> I do agree with this point, although as mentioned before this is not the first work that is entirely of this flavour (e.g. [1]) an important differentiating factor you offer however is a much more refined inner loop with finer guarantees compared to [1] ([1] simply does IS with no inner-loop and is thus very high variance). I think the presentation of the work would be improved and made much more clear if this differentiating factor (in particular the complexity of the inner loop scheme) is emphasized as the main contribution / differentiating factor, which also renders the method more practical compared to [1].

Thank you for your suggestion. We have modified our contribution part and focus on the differentiating factor. Moreover, we also try to address the differences in our related work part and Appendix.



> This said more high dimensional numerics (and complex dists) are needed to truly verify the method empirically. In particular, such verifications must be done such that ULA and the new proposed algorithm have roughly the same computation budget (e.g. inner_loop_steps * outer_loo_steps = ula_steps).

Thank you for your comments. We have added the high-dimensional empirical results in appendix F.3, which is similar to the low-dim case. The key reason is that dimension is not the leading factor, but the exponential log-Sobolev constant. So the behavior should be similar. And our computation complexity has already take the product (inner_loop_steps * outer_loo_steps) into account in both theory and experiments.

Reviewer comment (Experiments):
This MC-IS method for estimating the score will never work well in high dimensions due to variance, and thus works such as [1,2,3,4]—which are clearly aware of this formulation (as they either state it in their appendices or use it for subsequent calculation)—pursue an optimization alternative to estimating the drift. See [5] Figure 2 for how poor the performance of these estimators is compared to LMC and alternative NN approaches for learning the score as dimension scales. The authors briefly allude to this issue and propose combining with an inner ULA loop; however, as before, the authors missed that previous and more practical approaches (based on score matching) have already been developed to address this same issue. Without any empirical validation and careful comparison, it is very unclear why one would select the proposed approach.

Author response:
> There is a "diffusion models for sampling" aspect to your algorithm, and whilst prior work is mostly (not entirely e.g. [1]) focused on VI approaches to estimate the score, the overal should be discussed and prior relevant formulations of the score should be credited appropiately.


A key aspect of our work is the exploration of log-Sobolev constant dependency of the gradient complexity, where we have found that rdMC outperform conventional methods. This distinction makes a direct comparison with parameterized models infeasible. We have added discussion in Related work and Appendix A4.


> This is true but notice that prior work [2] establishes quite detailed bounds on the error that these parametric estimators of the score can potentially achieve, and it may be interesting (yet not necessary) to discuss at a high level how the complexity of these bounds compares to the results on the inner loop you have provided.

It is interesting to see the parametric error analysis of diffusion-based sampling algorithms. From a high-level perspective, [2] focus on the expressiveness of neural networks to learn the score function and we focus on the feasiblity and complexity to learn the score function for different time step $t$. Given our algorithm and analysis, if the neural network is well-specified (contains the real score function), the score learning of neural networks can be reformulated as a a statistical problem, where samples are obtained with computation efforts as we mentioned.




> I do agree with this point, although as mentioned before this is not the first work that is entirely of this flavour (e.g. [1]) an important differentiating factor you offer however is a much more refined inner loop with finer guarantees compared to [1] ([1] simply does IS with no inner-loop and is thus very high variance). I think the presentation of the work would be improved and made much more clear if this differentiating factor (in particular the complexity of the inner loop scheme) is emphasized as the main contribution / differentiating factor, which also renders the method more practical compared to [1].

Thank you for your suggestion. We have modified our contribution part and focus on the differentiating factor. Moreover, we also try to address the differences in our related work part and Appendix.



> This said more high dimensional numerics (and complex dists) are needed to truly verify the method empirically. In particular, such verifications must be done such that ULA and the new proposed algorithm have roughly the same computation budget (e.g. inner_loop_steps * outer_loo_steps = ula_steps).

Thank you for your comments. We have added the high-dimensional empirical results in appendix F.3, which is similar to the low-dim case. The key reason is that dimension is not the leading factor, but the exponential log-Sobolev constant. So the behavior should be similar. And our computation complexity has already take the product (inner_loop_steps * outer_loo_steps) into account in both theory and experiments.

Reviewer comment (Writing):
The assumptions in the work have been refined and the sketches are clearer to read in the revised versions.

Author response:
Thank you for your helpful comments. We have highlighted the “lower isoperimetric dependency” in Section 4.1.  Moreover, the parts related to Pinsker's inequality were rearranged for better reading experience.

For the Lipschitz continuity of the drift term, we add more explanations in our Appendix. We summarize it as below 

The main reason is that the delta distribution side makes $p_t$ ill-behaved (when the variance is approaching 0, which makes $-\log p_t \approx \Vert x\Vert^2/(2\sigma^2)$ is exploding). They also provided a sufficient condition when both $p_*\cdot\exp(\Vert x\Vert^2)$ and its gradient are Lipschitz continuous, and the former is bounded below by a positive value. However, this condition may not be met when the variance of $p_*$ exceeds 1, limiting its general applicability of this condition.
Thus, the generality of SFS remains an open question.
On the other hand, due to the smoothness of $p_\infty$ in OU process, the smoothness of $\log p_t$ is widely accepted in diffusion analysis [1,2]. [3] also suggest that the smoothness of concave $\log p_0$ can imply the smoothness of $\log p_t$. In summary, the smoothness of $\log p_t$ in OU process is much weaker than a process towards to a delta distribution, which generalize the analysis to more distributions.




[1] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint, arXiv:2209.11215, 2022a.

[2] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pp. 4735–4763. PMLR, 2023.

[3] Holden Lee, Chirag Pabbaraju, Anish Sevekari, and Andrej Risteski. Universal approximation for log-concave distributions using well-conditioned normalizing flows. arXiv preprint arXiv:2107.02951, 2021a.

Reviewer comment (Experiments):
Lack of practical and experimental results for complex distributions e.g. high-dimensional multimodal distributions. The method's performance is based on empirical observations and may not generalize well across diverse datasets or problem domains. Its effectiveness might be limited to specific scenarios and may not be universally applicable.

Author response:
Thanks. We have added the experiments for high-dimensional setting in our Appendix F3. In general, as the leading factor of isolated multi-mode distribution is the log-Sobolev constant (exponential dependency), the dimension dependency (polynomial) is not the main challenge. Thus, our algorithm still has significant improvement.

Reviewer comment (Theory):
Lack of complexity analysis. The combination of different sampling techniques (importance sampling, ULA) adds algorithmic complexity. Managing the interactions between these techniques and ensuring their proper integration can be challenging. Also, the method might demand significant computational resources, especially when dealing with large sample sizes and high-dimensional spaces. This could limit its practicality for resource-constrained applications. The sample size required for accurate estimation scales exponentially with the dimension due to the KL divergence between distributions. This exponential growth can make the method computationally infeasible for high-dimensional spaces. Could the authors please elaborate on these issues?

Author response:
Our paper provides a detailed complexity analysis in both Sections 4.1 and 4.2 under different assumptions. We incorporate both gradient complexity and sample complexity within our propositions, directly stemming from the log-Sobolev constant estimation and Theorem 1. 

**We highlight that our theory indicate that considering inner * outer loop, our algorithm is still much more efficient than Langevin-based algorithms. Our experimental results also verify this point.**

Additionally, it is noteworthy that our algorithm's dependency on the dimension $ d $ is polynomial, not exponential, further underscoring its efficiency. And our algorithm can reduce the exponential term induced by the log-Sobolev constant

Reviewer comment (Theory):
The accuracy of the estimation relies heavily on the dimensionality of the problem. High-dimensional spaces exacerbate the sample size requirement, making it challenging to apply the method effectively in real-world applications.

Author response:
Our paper provides a detailed complexity analysis in both Sections 4.1 and 4.2 under different assumptions. We incorporate both gradient complexity and sample complexity within our propositions, directly stemming from the log-Sobolev constant estimation and Theorem 1. 

**We highlight that our theory indicate that considering inner * outer loop, our algorithm is still much more efficient than Langevin-based algorithms. Our experimental results also verify this point.**

Additionally, it is noteworthy that our algorithm's dependency on the dimension $ d $ is polynomial, not exponential, further underscoring its efficiency. And our algorithm can reduce the exponential term induced by the log-Sobolev constant

Reviewer comment (Experiments):
Creating n Monte Carlo samples at each iteration can be computationally expensive, especially if n is large. This might limit the method's scalability and efficiency for high-dimensional or complex distributions.

Author response:
Thanks for your suggestion. In our analysis, we have already considered the complexity of $n$ samples. The main point of our algorithm is that even with $n$ samples, the total complexity is still less than Langevin-based algorithm. Note that the complexity with respect to $n$ is linear and the needed accuracy is polynomial, but the complexity of the original problem is exponential. 
In addition, the computation of MC samples can be paralled, while exponetial iterations cannot be reduced.

Reviewer comment (Theory):
The method uses random samples $\xi$ at each iteration. The quality of these random samples is crucial; if they are not truly random or are biased in some way, it can introduce errors in the sampled results. Furthermore, the way the samples are generated and combined in the update equation (step 6) could introduce bias if not done correctly. Biased estimators can lead to incorrect conclusions about the target distribution.

Author response:
Thank you for you question. Indeed, the quality of these random samples is crucial, so we analysis the necessary accuracy of the inner loop to obtain the proper samples in our Theorem 1. When the error is controlled, we can guarantee the convergence of our algorithm. In real practice, the algorithm is even less sensitive to the inner loop error, as the theoretical bound considers the worst case.

Reviewer comment (Theory):
The method's performance might degrade in high-dimensional spaces. Monte Carlo methods often face challenges in high-dimensional settings due to the curse of dimensionality, where the sampling space becomes sparse, making it harder to obtain representative samples.

Author response:
We appreciate your feedback. In response, we have included high-dimensional experiments in Appendix F3. It is crucial to note that our approach maintains a polynomial dependency on dimensionality, ensuring the identification of representative samples within a finite timeframe.

Reviewer comment (Experiments):
How sensitive is this framework to initial distribution $p_0$?

Author response:
Thank you for your question. Are you referring $\tilde{p}_0$? If so, we have added a discussion in the current Appendix F4.

Reviewer comment (Writing):
It is quite hard work to verify the theory. I would expect nothing less from such a paper, so by itself, it is, of course, not a problem. However, I believe there is room for improving the clarity and flow of the proofs.

Author response:
Thank you for your suggestion. We have significantly rearranged the proof and lemmas for better reading experience. Moreover, we have added a proof sketch in Appendix B. Please take a look and feel free to tell us if you have any question.

Reviewer comment (Reproducibility):
With the current presentation of the results, it is hard to verify the reproducibility of the results; in the experiments, the robustness of the algorithms to the input hyperparameters (for example, the choice of step size \eta in Algorithms 1/2).

Author response:
We have included the hyperparameters in our appendix and added the hyperlink in the main context.

Reviewer comment (Theory):
Lemma 9: not an obvious mismatch between the statement of the lemma and the final line in the proof (RHS of inequality is d/mu for the former, 1/mu for the latter).

Author response:
For Lemma 9 (previous notation, current Lemma 22), sorry for the confusion. The previous version of last line is for a single dimension. We have also added the reference for LSI and PI.

Reviewer comment (Theory):
The proof of lemma 9 starts with “It is known that LSI implies Poincare inequality with the same constant,…”. Perhaps a reference would help.

Author response:
For Lemma 9 (previous notation, current Lemma 22), sorry for the confusion. The previous version of last line is for a single dimension. We have also added the reference for LSI and PI.

Reviewer comment (Writing):
Unclear writing and claims not supported rigorously. See more below in the questions part.

Author response:
**Thank you for your suggestion. We followed your points and try to make our claims more clear and rigorous. We also highlight our analysis of $q$'s  log-Sobolev are in Sec 4.1 and 4.2. We try our best to revise our manuscript to address your concerns.**

**If you have any further inquiries or suggestions, please do not hesitate to reach out to us. Detailed response are as below.**

> *Q1. The rigorousness of the claims.*

Thank you for your suggestion. 
We have revised the claims as below:


1. Thus, we analyze the complexity to estimate the score by drawing samples from the posterior distribution and found that our proposed algorithm is much more efficient in certain cases when the log-Sobolev constant of the target distribution is small. 


2. For any $ t > 0 $, we observe that $ -\log q_{T-t} $ incorporates an additional quadratic term. In scenarios where $ p_* $ adheres to a log-Sobolev condition, this term enhances $ q_{T-t} $'s log-Sobolev constant, thereby accelerating convergence. Conversely, with heavy-tailed $ p_* $ (where $ f_* $'s growth is slower than a quadratic function), the extra term retains quadratic growth, yielding sub-Gaussian tails and log-Sobolev properties. Notably, as $ t $ approaches $ T $, the quadratic component becomes predominant, rendering $ q_{T-t} $ strongly log-concave and facilitating sampling.
In summary, every $ q_{T-t} $ exhibits a larger log-Sobolev constant than $ p_* $. As $ t $ increases, this constant grows, ultimately leading $ q_{T-t} $ towards strong convexity. Consequently, this provides a sequence of distributions with log-Sobolev constants surpassing those of $ p_* $, enabling efficient score estimation for $ \nabla\ln p_{T-t} $.

Reviewer comment (Theory):
At multiple points, the paper claims that the SDE resulting from the diffusion approach has better behaviour, e.g., the last sentence of Section 2.1 claims that the isoperimetric constant of this SDE is better. Right after Lemma 1, another claim is made "It is important to point out that the property of $q_{T-t}(\cdot | x)$ is better than $p_*$". Here as well, the sentence is badly written (what property?) But in any case, these claims, as far as I am able to see are not rigorously proven.

Author response:
**Thank you for your suggestion. We followed your points and try to make our claims more clear and rigorous. We also highlight our analysis of $q$'s  log-Sobolev are in Sec 4.1 and 4.2. We try our best to revise our manuscript to address your concerns.**

**If you have any further inquiries or suggestions, please do not hesitate to reach out to us. Detailed response are as below.**

> *Q1. The rigorousness of the claims.*

Thank you for your suggestion. 
We have revised the claims as below:


1. Thus, we analyze the complexity to estimate the score by drawing samples from the posterior distribution and found that our proposed algorithm is much more efficient in certain cases when the log-Sobolev constant of the target distribution is small. 


2. For any $ t > 0 $, we observe that $ -\log q_{T-t} $ incorporates an additional quadratic term. In scenarios where $ p_* $ adheres to a log-Sobolev condition, this term enhances $ q_{T-t} $'s log-Sobolev constant, thereby accelerating convergence. Conversely, with heavy-tailed $ p_* $ (where $ f_* $'s growth is slower than a quadratic function), the extra term retains quadratic growth, yielding sub-Gaussian tails and log-Sobolev properties. Notably, as $ t $ approaches $ T $, the quadratic component becomes predominant, rendering $ q_{T-t} $ strongly log-concave and facilitating sampling.
In summary, every $ q_{T-t} $ exhibits a larger log-Sobolev constant than $ p_* $. As $ t $ increases, this constant grows, ultimately leading $ q_{T-t} $ towards strong convexity. Consequently, this provides a sequence of distributions with log-Sobolev constants surpassing those of $ p_* $, enabling efficient score estimation for $ \nabla\ln p_{T-t} $.

Reviewer comment (Theory):
Theorem 1 *assumes* that $q_{T-{k\eta}}$ is log-Sobolev, instead of proving something about it. As such I think the whole motivation is unclear as authors didn't show how ill-posedness is tackled by reverse diffusion approach. Can authors show, if $p_*$ has a log-Sobolev constant, then $q$ does actually have a better behaviour in terms of this constant?

Author response:
> *Q2. log-Sobolev constant of $q$.*

Thank you for pointing out this. In fact, we are unable to whether $q_{T-k\eta}$ is log-Sobolev with A1 and A2.
That is the reason why we add Section 4.1 and 4.2 to discuss the assumption of $p_*$ and log-Sobolev constant of $q_{T-k\eta}$. We have added more explanation before 4.1 to highlight this point:

 *Note that the log-Sobolev constants of $q_{T-k\eta}$ depend on the properties of $p_ \ast$, so we should add more assumptions to estimate the log-Sobolev constants for $q_{T-k\eta}$. Next, we will demonstrate the benefits of using $q_{T-k\eta}$ for ill-conditioned LSI or non-LSI  $p_ \ast $.*

Reviewer comment (Presentation):
In Lemma 1, point out where the proof is in Appendix.

Author response:
> *Q3. Point out where the proof is in Appendix*

Thank you for your suggestion. We have revised our manuscript.