Paper: Prompt-tuning Latent Diffusion Models for Inverse Problems (ICLR-2024)

Reviewer comment (Writing):
The writing is poor and the submission is hard to follow.

Author response:
**A.** The main focus of this work was to propose a first method that leverages text-prompt embedding for solving inverse problems using latent diffusion models. Moreover, we have improved our manuscript also in the theoretical side from an alternating minimization perspective for a better understanding of why our method works better, and on which component our algorithm is improving the previous solvers.

1. Optimizing the text embedding can be thought of as corrections that are made on the posterior PF-ODE trajectory of reverse diffusion that modifies the embedding to meet the measurement conditions. Under this view, we do not need any assumptions on the independence between the noisy latents and the embedding.

2. The Encoder range space projection can be understood as performing MAP optimization under the VAE prior. Per this view, P2L can be seen as the first to leverage both the diffusion and the VAE generative prior into the posterior sampling scheme, further highlighting the strength of the approach.

These modifications offer better understanding on how the method can be derived and understood, and what type of approximation is used, and similarity and differences from the existing LDIS approaches such as PSLD. In this paper, the focus is more on the proof of concept that prompt tuning can yield improved performance, and that we can achieve better results by incorporating the prior learned through the VAE.

We have updated Section 3. We would appreciate it if the reviewer can point us to any specific section that the reviewers found it unclear or hard to follow.

Reviewer comment (Novelty):
The proposed projection is similar to that proposed in Chung et al. (2023b). Why the proposed approach can provide a good regularization is not clearly elaborated and why this approach is named projection is not clear.

Author response:
**A.** The proposed projection is completely irrelevant to DPS [1]. Note that in [1] there is no VAE or latent diffusion model, nor no exploitation of the text prompt. The two main contributions of this work are: 1. Exploit the text dependency in inverse problems by jointly optimizing the text representation; 2. Exploit the VAE prior to making the framework work on high-dimensions (through better latent diffusion models).

Regarding the specific proposed projection using the VAE prior, we elaborated on why the proposed projection approach yields good regularization in Section 3.2. The proposed method is called projection, as it clearly projects the latents onto the range space of the encoder.

Reviewer comment (Theory):
No further analysis on why the proposed projection can target any resolution and only subjective descriptions are proposed.

Author response:
**A.** Thanks for pointing this out. We have modified Appendix F to fully elaborate on the analysis. The proposed approach is useful for high-resolution image restoration because we are fully leveraging the power of the pre-trained VAE, and because these VAEs are much more robust to the change in the input resolution. In our method, we are proposing a method that can fully utilize this property, by alternatively optimizing latent and pixel space (through the VAE). In the revised version we have incorporated a theoretical analysis where we leverage the VAE prior within the formulation (now pointed out in Section 3.2).

Reviewer comment (Experiments):
The practicality of the proposed method is limited. The proposed P2L appears to be significantly slow. It necessitates 1000 reverse diffusion steps to achieve the results presented in the paper, each consisting of K backward propagations (BP) through the LDM for prompt tuning and 1 BP for a DPS-like step. It means that P2L could potentially be K times slower than DPS.

Author response:
**A.** Please see general comment C2. We agree that P2L requires many iterations. However, we would like to emphasize that P2L is more a proof of concept that optimizing text-prompt embeddings while solving inverse problems can lead to significantly better results. We believe that future investigations will yield methods that can build upon P2L to achieve solvers that are faster. Also, we would like to mention that there are certain use cases that do not require a fast solver but they require to reach the best possible quality (For example, one could use a "slow" method to generate the best possible result on on real data – where we don't know the ground-truth or reference, and later train a supervised method using the paired data).

Reviewer comment (Experiments):
The lack of experiments on the commonly used datasets of size 256x256 provides less evidence to support that the proposed prompt tuning is useful. The performance of the compared baselines on datasets of size 512x512 reported in this paper is unreliable. For example, there is a substantial degradation in performance for DIS methods, which may attribute to the high sensitivity to hyperparameters of these methods. I suggest the authors to add experiments on datasets of size 256x256 for fair comparison.

Author response:
**A.** We respectfully disagree. When running all DIS baselines, we did careful hyper-parameter tuning to yield the best possible result out of all methods. The reason for DIS underperforming on 512$\times$512 resolution is because pixel diffusion models that directly target this resolution are often low quality. This is why latent diffusion models and cascaded models dominate the mainstream when targeting image sizes beyond 256$\times$256. In this work, we specifically focus on solving inverse problems on higher resolutions beyond 256$\times$256, which is why we proceed on the latent space (i.e., Stable Diffusion).

In all prior works that tried to leverage latent diffusion models as priors for solving inverse problems [2,3,4], some experiments involve newly training a VAE+LDM from scratch for 256$\times$256 resolution, specified for a dataset that they are considering. This is not what we aim for, as explicitly pointed out in the last paragraph of Section 1: we are interested in using a foundational generative model that can operate on a fully general domain.

Reviewer comment (Experiments):
Comparison with the DIS method termed PGDM [R1] should be included. It does not require extensive hyperparameter tuning. Additional comparison with it on datasets of size 512x512 would be beneficial to demonstrate the strength of the proposed method. [R1] Song, Jiaming, et al. "Pseudoinverse-guided diffusion models for inverse problems." International Conference on Learning Representations. 2022.

Author response:
**A.** Please see modified Table 2, 8. We additionally compare against $\Pi$GDM [5].

Reviewer comment (Experiments):
In Section 3.2, the authors conduct an experiment using PSLD to show that it always diverges even if it started from a clean image. I believe that this experiment does not offer any insights because of two reasons: (i) The approximation used is not what was proposed in PSLD. In fact, the authors of PSLD show that aiming for any fixed point is not a good idea. Instead, they prove that the gluing objective helps recover the unique fixed point that exhibits contraction towards the optimal solution in a linear setting (Theorem 3.6). (ii) The fixed point solver works only for a unique step size (Theorem 3.5). Therefore, it is hard to get any meaningful conclusion from this experiment unless the step size is properly tuned or gluing objective is used instead.

Author response:
**W1. Relation to PSLD [1]**

**A.** Thank you for the thoughtful comment! We would like to clarify the experiment conducted in section 3.2. in the original submission. In this experiment, there is no *diffusion* component. We only iterate the encoding-decoding using the VAE used in Stable Diffusion v1.4. Hence, the experiment is *not* showing that PSLD always diverge. In fact, there is no inverse problem here, just the repeated application of the VAE encoder and decoder. Even when additionally using the gluing objective in between the encoding-decoding procedure, we still found that the encoding-decoding does not provide a fixed point solution. That being said, we also agree with the reviewer’s point on potential miscommunication to the readers, so we have removed the section in the revised version.

Reviewer comment (Experiments):
Choice of $\tau$: It is obvious that multiple steps of gradient descent would lead to better results than a single step. To make a fair comparison, the running time should be compared. In addition, there is another optimization problem being solved at every step of diffusion to get the near-optimal embeddings.

Author response:
**W3. Choice of $\tau$? Text-prompt embedding is solved by multiple-step optimization**

**A.** Thank you for pointing this out. Indeed, in prompt embedding optimization, we are solving an optimization problem rather than taking a single gradient step. We modified section 3.1. to give a better explanation about the method. Please consult general comment C1. Also, we include the comparison in the compute time for each algorithm in our manuscript (Table 5).

Reviewer comment (Theory):
The description about PSLD in the paragraph below eq. (6) is not fully correct. PSLD shows that there exists a step size for which guiding towards the fixed point is a good idea. Not for any step size.

Author response:
**W4. Imprecise description about PSLD [1]**

**A.** Thanks for pointing this out. We have modified the description to be more precise.

Reviewer comment (Theory):
How do you go from equation (11) to (12)? It is assumed that $C$ and $z_t$ are independent. However, this assumption is not properly justified.

Author response:
**Q1. Assumption on independence**

**A.** In the revised version we have removed this assumption. Please see general comment C1 and modified section 3.1.

Reviewer comment (Novelty):
Footnote 3 on page 5 is not the finding of this paper. It was shown in prior works (e.g. PSLD) that naively scaling DPS does not work.

Author response:
**Q2. Footnote 3 is not the finding of this paper**

**A.** The footnote was used for emphasizing that using the DPS for pixel- and latent- diffusion models lead to different consequences, and not to propose a new idea. That said, we agree that this may be redundant. The footnote was removed.

Reviewer comment (Experiments):
How many $C$ updates per $z_t$ update is needed? How does that affect the number of neural function evaluations? This should be discussed properly in the main draft.

Author response:
**Q3. How many $\mathcal{C}$ per $\mathbf{z}_t$ update? Runtime analysis needed**

**A.** Please see general comment 2. We now discuss the compute time that scales with the number of update steps used for $\mathcal{C}$ in the main paper. Please see Table 5.

Reviewer comment (Reproducibility):
The use of some parameters like K is not clearly shown in Algorithm 2. It should be used as an argument to the optimization block.

Author response:
**Q4. $K$ should be inside the optimization block**

**A.** Corrected. We appreciate the careful reading!

Reviewer comment (Theory):
The paper lacks a theoretical analysis of, for example, convergence.

Author response:
**A.** Please see our general response C1. While we limit the scope of our work and do not consider analysis of convergence, we have significantly extended our explanations on the theoretical motivations of P2L using an alternating minimization framework and clarified the approximation at each step. We also clarified the link to the existing LDIS approach such as PSLD. We believe that this extension adds much value to our manuscript, and we hope the reviewer also finds this interesting.

Reviewer comment (Evaluation):
From Table 1, the performance (PSNR) gains of P2L are subtle.

Author response:
**A.** We would like to respectfully disagree that the improvements are subtle. It is widely known that PSNR (distortion) and FID (perception) are two metrics that have trade-off [1,2]. When one opts for better PSNR, it leads to worse FID and vice versa. In this regard, the fact that our method keeps the PSNR level and improves FID by a large margin shows that P2L is actually pushing the pareto frontier forward. 

1. The performance gains are mostly seen in the FID score: *perceptual quality*. The information from the measurement may be sufficient for minimizing the *distortion*, but insufficient to achieve the best *perceptual* quality. The text embedding that we optimize for fills in this gap, as can be clearly seen in **-2.4, -0.9, -11.7,-4.0** improvements in the FID score.
2. The take-home message from Table 4 (now Table 3) is that the proposed components in the paper are *synergistic and provides a better trade-off*, and do not hamper each other. Moreover, similar to (1), the perceptual quality measured by FID does significantly increase.
3. The compute time required for latent diffusion model-based inverse problem solvers including P2L and all the baselines are almost perfectly linear on NFE. The difference between the compute time for using projection and proximal optimization is marginal. Moreover, while the difference between gluing and our proximal calibration may be not as pronounced on smaller noise levels, it gets more significant as we increase the noise level.

Reviewer comment (Experiments):
From the ablation experiments in Table 4, the difference between the results obtained by not using any of the three proposed modules and the results obtained by using all of them is not significant.

Author response:
1. The performance gains are mostly seen in the FID score: *perceptual quality*. The information from the measurement may be sufficient for minimizing the *distortion*, but insufficient to achieve the best *perceptual* quality. The text embedding that we optimize for fills in this gap, as can be clearly seen in **-2.4, -0.9, -11.7,-4.0** improvements in the FID score.
2. The take-home message from Table 4 (now Table 3) is that the proposed components in the paper are *synergistic and provides a better trade-off*, and do not hamper each other. Moreover, similar to (1), the perceptual quality measured by FID does significantly increase.
3. The compute time required for latent diffusion model-based inverse problem solvers including P2L and all the baselines are almost perfectly linear on NFE. The difference between the compute time for using projection and proximal optimization is marginal. Moreover, while the difference between gluing and our proximal calibration may be not as pronounced on smaller noise levels, it gets more significant as we increase the noise level.

Reviewer comment (Experiments):
From Table 5, the proposed proximal calibration is not that superior to the projection-based calibration, which is even cheaper and faster.

Author response:
1. The performance gains are mostly seen in the FID score: *perceptual quality*. The information from the measurement may be sufficient for minimizing the *distortion*, but insufficient to achieve the best *perceptual* quality. The text embedding that we optimize for fills in this gap, as can be clearly seen in **-2.4, -0.9, -11.7,-4.0** improvements in the FID score.
2. The take-home message from Table 4 (now Table 3) is that the proposed components in the paper are *synergistic and provides a better trade-off*, and do not hamper each other. Moreover, similar to (1), the perceptual quality measured by FID does significantly increase.
3. The compute time required for latent diffusion model-based inverse problem solvers including P2L and all the baselines are almost perfectly linear on NFE. The difference between the compute time for using projection and proximal optimization is marginal. Moreover, while the difference between gluing and our proximal calibration may be not as pronounced on smaller noise levels, it gets more significant as we increase the noise level.

Reviewer comment (Evaluation):
I'm curious if you have tried to figure out the text corresponding to the learned prompt. As shown in Table 1, including the methods presented in this paper and PALI's for prompt generation, what kind of textual information do they represent? I am curious about the interpretability of the prompts.

Author response:
**A.** Thank you for the thoughtful comment. Indeed, it would be very interesting and helpful if we could find out the “learned prompts” after the optimization. However, this is not possible as we leverage CLIP encoder, which does not have a corresponding decoding architecture. That said, using a diffusion model that leverages e.g. T-5 encoder which has a corresponding decoder, we believe that the learned prompts would be more explainable, which we leave as a future direction of research.

Reviewer comment (Presentation):
In algorithms 1-4 there are two functions called 'OptimizeEMB', which 'OptimizeEMB' do you use in algorithm 4?

Author response:
**A.** The ‘OptimizeEMB’ function is defined in a separate algorithm. In order to clarify this, we include Algorithm 2 in the appendix which corresponds to the definition of the ‘OptimizeEMB’ function with annotations. The same function is used for both Algorithm 1 and 3, which are the vanilla, and the adam versions of P2L.