Paper: Diffusion Sampling with Momentum for Mitigating Divergence Artifacts (ICLR-2024)

Reviewer comment (Experiments):
The paper primarily experiments with 10 or more generation steps, but it lacks analyses of extreme cases, such as one or two steps. It is suggested to evaluate the effectiveness of the proposed methods in these scenarios, for example, one or two steps.

Author response:
We appreciate the reviewer's suggestion to evaluate in extreme cases, particularly with one or two generation steps. Currently, every state-of-the-art high-order method relies on previous evaluations to better estimate the next evaluation point, and without a certain amount of previous evaluations, high-order methods will perform the same as low-order methods. In other words, all methods tend to yield similar results (see result in this [link](https://pic.in.th/image/TVMHCp)). Consequently, the performance of sampling becomes less dependent on the sampling method and more reliant on the model's effectiveness, especially when the number of steps is extremely low, such as one or two steps. So, when comparing with CM, any training-free methods will yield very similar, if not the same, result which can be seen in previous works [1,2].

Reviewer comment (Experiments):
The consistency model [1] performs well in one- and few-step generation. How effective is the method proposed in this paper compared with CM?

Author response:
We appreciate the reviewer's suggestion to evaluate in extreme cases, particularly with one or two generation steps. Currently, every state-of-the-art high-order method relies on previous evaluations to better estimate the next evaluation point, and without a certain amount of previous evaluations, high-order methods will perform the same as low-order methods. In other words, all methods tend to yield similar results (see result in this [link](https://pic.in.th/image/TVMHCp)). Consequently, the performance of sampling becomes less dependent on the sampling method and more reliant on the model's effectiveness, especially when the number of steps is extremely low, such as one or two steps. So, when comparing with CM, any training-free methods will yield very similar, if not the same, result which can be seen in previous works [1,2].  
It is true that methods like the consistency model can yield the extreme efficiency of one- or two-step generation. However, they require creating new models by either distilling old ones or training from scratch. This is contrary to our approach, which focuses on designing a training-free pipeline that can be applied to any existing models.

Reviewer comment (Experiments):
How effective are the proposed methods in extremely small generation steps, such as one or two?

Author response:
We appreciate the reviewer's suggestion to evaluate in extreme cases, particularly with one or two generation steps. Currently, every state-of-the-art high-order method relies on previous evaluations to better estimate the next evaluation point, and without a certain amount of previous evaluations, high-order methods will perform the same as low-order methods. In other words, all methods tend to yield similar results (see result in this [link](https://pic.in.th/image/TVMHCp)). Consequently, the performance of sampling becomes less dependent on the sampling method and more reliant on the model's effectiveness, especially when the number of steps is extremely low, such as one or two steps. So, when comparing with CM, any training-free methods will yield very similar, if not the same, result which can be seen in previous works [1,2].

Reviewer comment (Novelty):
The technical novelty behind this work seems to be not significant. The main techniques used in this paper are directly borrowed from Polyak’s heavy ball (HB) momentum method, a conventional optimization algorithm. Besides, the main improvements of this work are built based on DPM++ and PLMS.

Author response:
- Our GHVB, a high-order generalization of Polyak's heavy ball momentum method, is an original approach not borrowed from elsewhere. Though, we acknowledge that the HB (not GHVB) algorithm may be perceived as an application of existing concepts like momentum. Nonetheless, our distinct contribution is being the first to employ and analyze HB as an ODE solver within the context of diffusion sampling. Our study also provides novel findings resulting from extensive experiments and well-tested hypotheses, which serve as a basis for practical adoption  
- Additionally, our HB method may seem to build upon previous methods like DPM++ and PLMS because it is designed as an add-on specifically aimed at addressing their divergence artifact problem.

Reviewer comment (Experiments):
While two methods are proposed in the same paper, it is unclear which one should be used under what circumstances. The paper only gives some vague statements without comprehensive comparison.

Author response:
- A direct comparison is presented in Figure 5, demonstrating that both HB and GHVB effectively reduce artifacts. However, HB exhibits a faster accuracy drop, leading to blurry images.  A discussion between these methods can be found in Section 7, emphasizing HB's role as an add-on for artifact reduction, while GHVB offers flexibility in method selection. Therefore, we advocate for choosing the high-order generalization method (GHVB) over first-order methods like HB. Ideally, the selection should prioritize the highest-order method whose stability region still covers all eigenvalues, as detailed in Appendix S, Topic Q2 (moved to Appendix T in the revision).

Reviewer comment (Experiments):
While guided diffusion sampling is considered, the effectiveness of the HB/GHVB under different scaling factor “s” is not well discussed.

Author response:
- Please see Figures 24-32 in Appendix N (moved to Fig. 25-33 in Appendix O), where we demonstrate the effectiveness of our HB and GHVB under various guidance scales for guided diffusion sampling.

Reviewer comment (Evaluation):
While the authors mentioned that the problem setup is more challenging in this paper than previous works, it is unclear what the challenges are. More discussions about why PLMS and DPM-Solver++ perform worse than their original claims would strengthen this proposal.

Author response:
- A short answer: The challenges here refer to the ability to speed up and reduce the number of sampling steps while avoiding divergence artifacts. All these algorithms generate artifacts, but only when the sampling steps drop below a certain threshold, which might not always be tested or demonstrated in their papers. In theory, eigenvalues and step size are primary factors leading to divergence in ODE solving. Reducing the step size is the most obvious way to make the problem more challenging, as highlighted in Section 1's second paragraph and depicted in Figure 1. Another approach involves altering sampling problems' eigenvalues, such as those using guidance sampling, higher guidance scales, LoRA fine-tuning, ControlNet, or the "Reference-Only" pipelines detailed in Appendices M and N (moved to N and O). These sampling problems tend to produce more artifacts at the same number of sampling steps compared to standard diffusion sampling, making them more challenging in our context.

Reviewer comment (Evaluation):
Given that the 1000-Step DDIM’s sample is considered the benchmark, it would be reasonable to include evaluation metrics such as L2, LIPIS, and FID comparing HB/GHVB to DDIM, as depicted in Figure 11.

Author response:
- We agree with and appreciate your suggestion and will update the evaluation to use the 1000-step DDIM including Figure 11. According to the convergence theorem, numerical methods converge to the same exact solution with a high number of steps. Hence, this modification should not impact the final outcome.

Reviewer comment (Evaluation):
In Figure 12, the authors attribute the inconsistency of GHVB 2.5 and 3.5 to estimated error or other sources of error without further justifications. It would be helpful to discuss this more for better understanding.

Author response:
- The discrepancies observed in the plots depicted in Figure 12, which appear less consistent with the theoretical predictions, may come from various sources and reasons. One possible explanation is the limited precision of neural networks and the already very low errors for 640-step sampling of Figure 11 ($\sim 10^{-3}$). This scenario amplifies the sensitivity of the $e_{new} / e_{old}$ ratio to computational inaccuracies, even as small as $10^{-4}$.To illustrate this potential behavior, we demonstrate that errors from both the 999-step and 1,000-step DDIM can vary significantly by up to 0.0026. Theoretically, this difference should align much closer to zero. Additional sources of error include the time discretization nature of the diffusion network, which might not consistently across all number of steps (e.g., 320 vs 640). These errors are negligible when the total error is  large but can significantly compound and affect our *empirical approximation* of convergence order when dealing with very small errors.  
- [link to the experiment](https://colab.research.google.com/drive/11d5eXZHaG2qCJY_TgB8-zq1fagSmA7b8?usp=sharing)

Reviewer comment (Evaluation):
The comparisons and discussions between HB and GHVB are not sufficient in the paper’s current state. There is no clear cut which method is better for both conditional and unconditional diffusion sampling.

Author response:
- A direct comparison is presented in Figure 5, demonstrating that both HB and GHVB effectively reduce artifacts. However, HB exhibits a faster accuracy drop, leading to blurry images.  A discussion between these methods can be found in Section 7, emphasizing HB's role as an add-on for artifact reduction, while GHVB offers flexibility in method selection. Therefore, we advocate for choosing the high-order generalization method (GHVB) over first-order methods like HB. Ideally, the selection should prioritize the highest-order method whose stability region still covers all eigenvalues, as detailed in Appendix S, Topic Q2 (moved to Appendix T in the revision).

Reviewer comment (Experiments):
The proposed method should be compared with the state-of-art methods in reducing divergence artifacts if it is a big challenge in diffusion models.

Author response:
As far as we understand, no other researchers have identified divergence artifacts as a problem in diffusion sampling. Therefore, we solely compare our approach with the current state-of-the-art diffusion sampling techniques outlined in both our main paper and appendix.

Reviewer comment (Experiments):
The stated motivation is diffusion model acceleration. Experiments are limited in comparing the results of few-step sampling, lacking clear numerical experiments in model acceleration.

Author response:
We would like to first clarify the term 'acceleration’ in the context of our work. Here, acceleration refers to our ability to decrease the number of sampling steps while still producing good results without degeneration or divergence artifacts. For example, a sampling technique that produces artifacts using 10 sampling steps is considered slower than another that still remains artifact-free at step 10, when applied to the same diffusion model. In this context, the number of steps is a reasonable representation of speed, as the extra computation time for each step—due to computations related to momentum or other high-order aggregation—is negligible compared to the network evaluation time. A method that accelerates sampling by 25%, for example, means that the number of sampling steps can be reduced by 25% without the onset of artifacts."
- Regarding your concern, please see Appendix M (moved to N in the revision) where we have provided comprehensive analyses of speed improvement, some of which is reproduced here. In Appendix M.1 (N.1), our experiments demonstrate that HB and GHVB reduce sampling time by 16.53% and 25.38%, respectively, compared to PLMS4 without momentum. Moreover, DPM-Solver++ with HB momentum achieves an 18.22% reduction in sampling time compared to the same algorithm without momentum. In Appendix M.2 (N.2), our experiments indicate significant reductions in sampling time from PLMS4, amounting to 32.40% and 42.92% with HB and GHVB, respectively. We believe these quantitative results show that our algorithms yield clear sampling speedups.

Reviewer comment (Theory):
The main difference between HB and GHVB is that HB calculates the moving average after summing high-order coefficients, whereas GHVB calculates it before the summation. Why does such a difference lead a larger stability region?

Author response:
In the concluding paragraph of Section 2.2, we highlighted that higher-order methods tend to have smaller stability regions. Given HB's first-order convergence compared to GHVB's designed high-order convergence, HB naturally exhibits a larger stability region. To understand how formulation differences impact convergence orders, please refer to Section 4.2 and Appendix F (moved to G). Higher convergence orders indicate that more terms are matched in the Taylor expansion, resulting in a more accurate approximation. Notably, GHVB matches more terms than HB

Reviewer comment (Evaluation):
With additional training, what is the proposed methods’ complexity or cost?

Author response:
Our method is training-free and applicable to any diffusion sampling. We believe there might be some typos in the question. In case the reviewer meant the complexity and/or cost induced by our sampling algorithms, the inclusion of one line of code for computing the moving average has a very small impact on sampling time, as detailed in Table 12 of Appendix R (moved to S). Both of our methods show no significant differences in sampling times compared to other methods. We hope this clarifies these aspects.

Reviewer comment (Writing):
I recommend that authors add a paragraph explaining what "sampling" means in the context of diffusion in the appendix, so that the content can be more self-contained. From what I understand about the main text, authors mean generating/inferring an image with trained diffusion models. This is not equivalent to the meaning of illustrating the distribution of all potentially generated images given underlying diffusion models.

Author response:
Thank you for the valuable feedback. We appreciate the suggestions. We are happy to add a section in the appendix to explain more about diffusion sampling, create a table summarizing numerical methods for diffusion solver, and include a section with detailed information on diffusion training. (added to Appendix A in the revision)

Reviewer comment (Presentation):
I suggest that authors make a table to list all used numerical formats explicitly, either in the main text or in appendix, to generate images. In this way, readers can associate the listed methods in each table/figure with specific algorithms.

Author response:
Thank you for the valuable feedback. We appreciate the suggestions. We are happy to add a section in the appendix to explain more about diffusion sampling, create a table summarizing numerical methods for diffusion solver, and include a section with detailed information on diffusion training. (added to Appendix A in the revision)

Reviewer comment (Evaluation):
Are metrics "high-frequency error norm (HFEN)" [MR image reconstruction from highly undersampled k-space data by dictionary learning, Ravishankar and Bresler, 2011] and Structural Similarity Index (SSIM) potentially relevant to measure the divergence artifacts (section 5.1)? If yes, then reporting evaluation results in these two metrics can be helpful.

Author response:
In contrast to the magnitude score, the HFEN and SSIM metrics do not directly quantify the extent of divergence artifacts in the results. However, they can detect divergence by comparing the differences between individual samples and their convergence results, akin to the use of l2 and LPIPS metrics, which we have employed throughout our paper. We appreciate this suggestion and will include these measurements in our paper.

Reviewer comment (Writing):
Conceptually, I would like to understand better what authors mean by "classifier-guided diffusion sampling". What is the difference (conceptually and when it comes to implementation) between classifier-guidance and text-prompt based generation?

Author response:
We acknowledge the confusion and will address it by adding a section in the appendix to provide a clearer explanation about guided diffusion sampling.  
In short, classifier-guided diffusion sampling [1] utilizes another model, typically a classifier model, to guide the sampling process towards a desired direction. This is in contrast to classifier-free sampling [2], where the diffusion model itself directs the outcome. Text-to-Image generation can be achieved through both classifier-guided sampling, as seen in CLIP-guided sampling, or classifier-free sampling. However, in approaches like Stable Diffusion, the sampling is predominantly considered classifier-free.  

- [1] : Diffusion Models Beat GANs on Image Classification  
- [2] : Classifier-Free Diffusion Guidance