Paper: Manifold Preserving Guided Diffusion (ICLR-2024)

Reviewer comment (Evaluation):
The results of style guidance look not that appealing.

Author response:
**Better Results for Style Guidance Stable Diffusion Generation (Weakness 2):** Based on Reviewer Yt4g’s suggestion, we were able to further improve the performance of our method on style guidance Stable Diffusion generation, and we have included new qualitative showcases in Figures 20 and 21 and quantitative results in Tables 2 and 4 in our paper.

Reviewer comment (Experiments):
If possible, user study would make the experimental part stronger.

Author response:
**User Study (Weakness 3):** Thank you for your suggestion. We have conducted a user study for the style-guided Stable Diffusion generation task on Amazon Mechanical Turk to compare our method (MPGD-LDM) and two baselines (FreeDoM and LGD-MC). Details about the experiment are described in the appendix.

    The results are shown in Table 5 in the appendix as well as below. Users’ responses regarding Style and Text generally align with the style score and CLIP score in Table 2. In terms of overall user preference, our method outperformed both methods, which suggests that our method finds a better sweet spot balancing the text prompt condition and the style guidance.

|         Method        | Style Consistency (Win/Lose/Draw) | Prompt Consistency (Win/Lose/Draw) | Overall (Win/Lose/Draw) |
|:---------------------:|:---------------------------------:|:----------------------------------:|:-----------------------:|
| MPGD-LDM v.s. FreeDoM |             47%/45%/8%            |             32%/66%/2%             |        49%/45%/6%       |
|  MPGD-LDM v.s. LGD-MC |             27%/64%/9%            |             69%/29%/2%             |        53%/43%/4%       |

Reviewer comment (Writing):
There are some typos, e.g., in Eq. 2 (\alpha_t --> \alpha_{t-1}) and Fig. 14 (bitterfly --> butterfly).

Author response:
**Typos (Weakness 4):** Thank you for pointing out the typos. We have corrected these errors in the revised manuscript.

Reviewer comment (Evaluation):
The paper claimed that the proposed MPGD can generate images with 3.8 times speed up compared with the baseline. The main paper should give more details on why it can speed up with the same number of iterations?

Author response:
The reason for the speed-up is that our proposed MPGD does not require backpropagation through the diffusion model, which is generally known to be time-consuming. To be more specific, in DPS based methods, the guidance is calculated as $\nabla_{x_t} L(x_{0|t};y) = \nabla_{x_{0|t}} L(x_{0|t};y) \frac{\partial x_{0|t}}{\partial x_t}$ where $\frac{\partial x_{0|t}}{\partial x_t}$ requires backpropagation through the diffusion model. On the other hand, in MPGD, we only calculate the gradient $\nabla_{x_{0|t}} L(x_{0|t};y)$ with respect to $x_{0|t}$. Althoughin MPGD-AE and MPGD-Z, evaluating the autoencoder incurs additional costs, it is still faster than computing the gradients for the diffusion model itself. We have briefly discussed this reason in Section 4.2.1, and modified the writings in the experiment discussions to emphasize this reason.

Reviewer comment (Theory):
The theoretical analysis on the auto-encoder in theorem 2 says that the gradient of auto-encoder lies on the tangent space of manifold. The analysis relies on the linear manifold hypothesis. Is this assumption realistic for real image set, and how to understand that the image manifold is linear?

Author response:
In response to your question about the linear manifold hypothesis in our theorem 2 analysis, it's important to clarify that our analysis of "the gradient of the auto-encoder lying on the tangent space of the manifold" does not actually depend on the linear manifold hypothesis. Proposition 1, which was initially proposed by [Chung et al. 2022], is based on this hypothesis.

    While we acknowledge that the linear manifold hypothesis may not be entirely realistic for all complex real image datasets, it is often observed that this assumption holds locally. A practical example of this can be seen in image interpolation. Consider interpolating between two headshots of the same person, where one image features black hair and the other blond hair. The interpolation can result in another plausible headshot of the same person with brown hair. This example illustrates that even though the overall image manifold might not be linear, local linearity can be a reasonable approximation for certain kinds of image transformations.

    Many prior works in the literature also make similar assumptions and implications. For example, the classic facial recognition paper [Turk et al., 1991] has discussed the validity of linear manifold hypothesis for specific subsets of image data such as well-aligned human face images. Hence, we believe this assumption can be made for the purposes of our analysis and is also meaningful for practical applications of our proposed methods.

[Chung et al., 2022] Improving diffusion models for inverse problems using manifold constraints. NeurIPS 2022.

[Turk et al., 1991] “Eigenfaces for Recognition”, Journal of cognitive neuroscience.

Reviewer comment (Experiments):
In the experiments for super-resolution, there are several methods using diffusion model for image super-resolution or other restoration methods, e.g., Denoising diffusion null-space model, etc. How about the comparisons with these Sota diffusion-based image inverse models?

Author response:
Thank you for your suggestion. We have added the comparison experiment with DDNM on the FFHQ super-resolution task in Section E.2 in our paper. But before we dive into the discussion about the experiment, we would like to emphasize that DDNM is designed for only solving linear inverse problems, and it requires direct access to the operation matrix, its pseudo-inverse/SVD and the noise scale for the measurement, which are not parts of the assumptions that we have in our problem setting. As a result, DDNM is not applicable to the general setting of paper. Nevertheless, we also think it is valuable to better position our paper in the literature of linear inverse problem solving, and therefore we conducted the experiments described below.

    To make a fair comparison, we use the simplified version of DDNM with no time traveling and the same unconditional diffusion model pre-trained by the authors of DPS, which is a smaller model compared to the one DDNM used in their paper. Due to code availability, we use average pooling as the interpolation method which is different from our original experiment setting. Figure 11 demonstrates the qualitative results, Figure 12 shows the enlarged details and Figure 10 shows the quantitative results. As we can observe in the figures, the simplified version of DDNM can achieve an equally fast sampling speed compared to our method and can also obtain similar guidance quality. However, the images generated from DDNM exhibit various artifacts, such as high-frequency circular patterns and overly smoothed generation. These artifacts prevent DDNM from maintaining high fidelity while our method can generate more realistic details. 

    That being said, we would like to point out that the generated images from DDNM tend to maintain better shapes whereas our method hallucinates small details more than DDNM. Therefore, when we compare PSNR values (Table 3), we can observe that DDNM is significantly better than our method. We also acknowledge that in the original paper of DDNM, in order to solve the noisy inverse problems, the authors suggest using multi-step time traveling to improve the performance which we did not deploy in order to make a fair comparison in terms of run time. Therefore, we believe that DDNM still has certain advantages over our method in terms of solving specific inverse problems. We think incorporating the consistency constraint in DDNM with our method can potentially achieve the best of both worlds, and will be an interesting future work direction.

Reviewer comment (Experiments):
The experiments do not clearly demonstrate the idea of guiding the sampling process with an off-the-shelf network. In all three experiments, a reference image is used to guide the sampling. The projection to the manifold in these cases is shown to successfully guide the diffusion towards an image in the reference image neighborhood. However, it is not clear how that would work in the case where the guidance is given by a more general differentiable loss $L$. This raises the question of whether the proposed method is applicable in cases other than the limited set of inverse problems presented, where the guidance may be less informative. An example would be an $L$ that measures the log-likelihood of an attribute in the image (e.g. blonde hair).

Author response:
1. **More General Guidance Loss w/o Reference Images (Weakness 1 & Question 2):** Thank you for bringing up this question. We would like to first clarify that our method is NOT limited to tasks with reference images, and therefore, we really appreciate you asking this question because although two of our original choices of tasks (FaceID guidance and style guidance generation) are not inverse problems, your question helped us realize that the previous example tasks we chose happen to all involve reference images. As a result, we have added an additional experiment where we use the CLIP model and text prompt to guide the pixel-space CelebA-HQ model. In Section E.1 and Figure 9 in the appendix, we show examples of samples guided by the text prompt and our method in comparison with unconditional DDIM samples of the same random seed. Our method is able to create images that follow the text description provided while maintaining high fidelity using an unconditionally pre-trained diffusion model.

Reviewer comment (Evaluation):
The qualitative results in Figure 6 raise questions about the diversity of the samples drawn. The generated images shown are almost all identical and deviate from the distribution of images in the original CelebA dataset (i.e. the colors are completely unnatural). The baseline also performs badly but the distribution seems closer to the original $p(x)$ that the model learned.

Author response:
2. **Diversity of the Generated Images (Weakness 2):** We apologize for the confusion caused by the presentation of this figure. Firstly, we would like to respectfully point out that the DDIM samples are the ones that showcase “the original p(x) that the model learned”. Therefore, in Figure 6 we can observe that our method is actually closer to the DDIM samples (and therefore the original p(x) that the model learned) regarding the color, structure, and other fine-grained details in the images. In fact, this is precisely because our method performs on the tangent spaces of the original DDIM clean data estimations, and therefore, is able to stay close to the original distribution while performing guided generation. We have also confirmed these qualitative observations with quantitative analysis in Table 1. The reason why these images seem to have strange color schemes is because the DDIM trajectories happen to produce these samples. We can also observe that FreeDoM, which is one of our baselines, generates similar images. 

    That being said, to demonstrate our method’s generation diversity, we have included Figure 16 and an additional Figure 17 in the appendix for the pixel space FaceID guidance generation experiment. In these figures, we demonstrate that images generated by our method show diversity in various aspects, such as color, style, and facial expression.

Reviewer comment (Presentation):
The related works section should be in the main paper. This is necessary to draw clear distinctions between the existing works and the proposed approach and highlight what the authors' main contributions are.

Author response:
3. **Related Works (Weakness 3):** Thank you for your feedback. We initially structured our paper to address the most relevant prior works and their limitations in Sections 2 and 3 before introducing our method in order to adhere to the page constraints while reserving an in-depth exploration of further related literature for the appendix. Based on your suggestion, we have now added Section 2.4 in our main paper to briefly discuss more related works while keeping the detailed discussion in the appendix.

Reviewer comment (Reproducibility):
Could the authors clarify what the $L$ function that is used in the different experiments is?

Author response:
4. **Specifications of loss functions used in experiments (Question 1):** We apologize for the confusion. Regarding the style-guided generation task, we recognized that our definition of the loss function was not as clear as it should have been. To rectify this, we have provided additional details in Appendix D for further clarification. To summarize, the settings for the super-resolution task are the same as those used in DPS [Chung et al., 2023], and the settings for the FaceID guidance generation and text-to-image style guidance generation are the same as those employed in FreeDoM [Yu et al., 2023].

[Chung et al., 2023] “Diffusion posterior sampling for general noisy inverse problems”, In Proc. ICLR, 2023.

[Yu et al., 2023] “FreeDoM: Training-free energy-guided conditional diffusion model”, In Proc. CVPR, 2023.

Reviewer comment (Experiments):
Can the authors showcase their method on an experiment that is not based on sampling with a reference image? Is this a limitation of their method or can it be generalized to other formulations of $L$?

Author response:
1. **More General Guidance Loss w/o Reference Images (Weakness 1 & Question 2):** Thank you for bringing up this question. We would like to first clarify that our method is NOT limited to tasks with reference images, and therefore, we really appreciate you asking this question because although two of our original choices of tasks (FaceID guidance and style guidance generation) are not inverse problems, your question helped us realize that the previous example tasks we chose happen to all involve reference images. As a result, we have added an additional experiment where we use the CLIP model and text prompt to guide the pixel-space CelebA-HQ model. In Section E.1 and Figure 9 in the appendix, we show examples of samples guided by the text prompt and our method in comparison with unconditional DDIM samples of the same random seed. Our method is able to create images that follow the text description provided while maintaining high fidelity using an unconditionally pre-trained diffusion model.

Reviewer comment (Evaluation):
While the paper appropriately utilizes KID as a less biased metric, it could benefit from including FID (Fréchet Inception Distance) as a more widely recognized and standardized metric for related work comparison. The absence of FID makes it challenging for readers to assess the method's performance in relation to existing or future approaches.

Author response:
**FID (Weakness 1):** We agree that it would be helpful to include FID since it is a more popular metric in the community. However, we would like to emphasize that for FID to be reliable, it would require at least 10k samples to evaluate for each experiment [Bińkowski et al., 2018]. Given the limited timeframe of the rebuttal period and the required sampling time for all the methods compared, it is not very feasible for us to produce a large enough number of samples to make proper FID comparisons, and we respectfully propose deferring this extensive analysis to future work, to ensure thorough and accurate evaluation.

Reviewer comment (Evaluation):
The paper misses the opportunity to provide insight into failure cases when it comes to text-guided generation. This is particularly important given its widespread use in the community.

Author response:
**Additional Failure Cases (Weakness 2):** Based on your comment, we have added some noteworthy failure cases from the images generated in the style transfer task with StableDiffusion and have included a discussion in the Appendix F.2. In summary, there exist some instances where the expected style is not reflected in the generated images due to the design of the loss function or the relationship between the provided prompt and the reference style.

Reviewer comment (Evaluation):
Is there any behavior change in the classifier-free guidance scale for text in the style guided experiments?

Author response:
**Effect of Classifier-Free Guidance Scale (Question 2):** Thank you for this question. In the appendix E.4 and the table below, we have included an additional quantitative comparison in style guided Stable Diffusion generation with various classifier-free guidance scales (CFG). This analysis actually helped us to produce better results with our method in this experiment, and we have also modified our main paper to include the updated results with further hyperparameter tuning. It's worth noting that the CFG scale has a positive impact on the style score, and strong CFG scales appear to help decrease the loss function. However, although in vanilla text-to-image generation tasks a larger CFG scale tends to lead to a higher CLIP score [Nichol et al., 2022], since the distribution we aim to sample from is also guided by another external loss function, we do not observe the same trend in our setting. In practice, we would suggest our users to adjust this hyperparameter to suit their preference of tradeoff between the style guidance and the text prompt condition.


|MPGD-LDM |**Style**$\downarrow$|**CLIP**$\uparrow$|
|----|----|----|
|CFG=2.5|493.5|26.98|
|CFG=5.0|459.8|**27.08**|
|CFG=7.5|**441.0**|26.61|


[Nichol et al., , 2022] “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models”, Proceedings of the ICML, 2022.