Paper: Solving Inverse Problem With Unspecified Forward Operator Using Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
Focusing on blind deblurring, UFODM becomes similar to FastDiffusionEM, which also uses $\Pi$GDM update step for x, and uses MAP to update the kernels. The difference is that [1] uses the plug-and-play denoiser prior augmented with expectation-maximization (EM) approach, which is usually superior to standard gradient descent. Given that the UFODM does not compare directly with [1], I can only guess that it will be inferior to FastDiffusionEM.

Author response:
W1, W2, W3, Q2: We appreciate your suggestion to compare UFODM with FastDiffusionEM. While UFODM and FastDiffusionEM share certain methodological elements, our approach is distinct in its focus on handling unspecified forward operators. In light of your feedback, we recognize the value in conducting a direct comparison with FastDiffusionEM and will consider incorporating this in our future work.
In terms of the comparisons drawn with BlindDPS and GibbsDDRM, it is essential to underscore that our primary objective is centered around scenarios involving unspecified forward operators. While attaining state-of-the-art performance on specific tasks is undoubtedly important, it was not the principal focus of our study. Our approach is designed to broaden the scope and applicability of solving inverse problems in more generalized settings.

Reviewer comment (Experiments):
Extending 1, I think the authors must compare against FastDiffusionEM. One might complain that it is only an arxiv paper yet, but considering the highly fast-growing and competitive nature of the field, and also given that the code is already open-sourced, it wouldn't take too long to run a head-to-head comparison against it.

Author response:
W1, W2, W3, Q2: We appreciate your suggestion to compare UFODM with FastDiffusionEM. While UFODM and FastDiffusionEM share certain methodological elements, our approach is distinct in its focus on handling unspecified forward operators. In light of your feedback, we recognize the value in conducting a direct comparison with FastDiffusionEM and will consider incorporating this in our future work.
In terms of the comparisons drawn with BlindDPS and GibbsDDRM, it is essential to underscore that our primary objective is centered around scenarios involving unspecified forward operators. While attaining state-of-the-art performance on specific tasks is undoubtedly important, it was not the principal focus of our study. Our approach is designed to broaden the scope and applicability of solving inverse problems in more generalized settings.

Reviewer comment (Experiments):
Even when comparing against BlindDPS [2] and GibbsDDRM [3], it does not seem that UFODM consistently outperforms the prior arts. Yes, it does outperform [2,3] when time-travel trick is utilized, but time-travel is a standard trick that can easily be incorporated into all of the solvers, which makes the comparison unfair. Given that UFODM is an ad-hoc mix of Gibbs sampling and MAP optimization, it is important to show the empirical strength, which, in my opinion, is insufficient.

Author response:
W1, W2, W3, Q2: We appreciate your suggestion to compare UFODM with FastDiffusionEM. While UFODM and FastDiffusionEM share certain methodological elements, our approach is distinct in its focus on handling unspecified forward operators. In light of your feedback, we recognize the value in conducting a direct comparison with FastDiffusionEM and will consider incorporating this in our future work.
In terms of the comparisons drawn with BlindDPS and GibbsDDRM, it is essential to underscore that our primary objective is centered around scenarios involving unspecified forward operators. While attaining state-of-the-art performance on specific tasks is undoubtedly important, it was not the principal focus of our study. Our approach is designed to broaden the scope and applicability of solving inverse problems in more generalized settings.

Reviewer comment (Theory):
The problem setting for unknown $\mathcal{H}$ is unrealistic. Considering even the canonical degradations that arise in computational photography, the number of classes in the set exponentially grows. This is especially true if we start to consider a mixture of degradations, which usually happen in the real-world.

Author response:
W4: Thank you for your insightful comment on the realism of our approach to unspecified H in complex problem settings. While we acknowledge that our current setting does not entirely capture the multifaceted nature of real-world scenarios, we believe it represents a meaningful step forward compared to traditional settings of inverse problems and blind inverse problems. Our approach, by reducing restrictions typically encountered in these conventional frameworks, allows for more flexible and comprehensive solutions. This adaptability is crucial in complex scenarios where standard methods might not be sufficient.
We provide a qualitative analysis in Figure 3, showcasing our method's adeptness in managing mixtures of two distinct degradations. This demonstration serves as a testament to the versatility of our approach. Our aim with this setting is not to fully replicate the intricacies of real-world degradations but to provide a more versatile foundation upon which further research can build. We envision future extensions of our work that will progressively incorporate the complexity of real-world scenarios.

Reviewer comment (Evaluation):
Even if the algorithm is not theoretically grounded, it is intuitive when one does not consider sampling from $\mathcal{H}$. However, the part of inferring $\mathcal{H}$ seems quite odd, or at least unrealistic. In order to choose $\mathcal{H}^*$, one has to run all the diffusion process for each candidate $\mathcal{H}$, which will be painfully slow as the number of possible class increases. Moreover, the criterion is the distance from the measurement, which does not seem to be a particularly *good* metric. My concern is corroborated from seeing that the accuracy of $\mathcal{H}$ for ImageNet is 61.3% even when there are only 3 classes to choose from.

Author response:
W5: We have employed a parallel sampling method within the diffusion process. Each forward function in this process corresponds to a single sample in a batch. This parallel sampling technique allows us to simultaneously estimate samples from all possible function forms represented in the set, thereby significantly enhancing the efficiency of the process. This method alleviates the issue of having to run the diffusion process separately for each candidate $\mathcal{H}$, ensuring a more practical and time-efficient approach.
Regarding the criterion based on the distance from the measurement, we recognize that this metric has its limitations and there is indeed room for improvement. We are committed to refining this aspect of our methodology in future work to develop a more robust and accurate criterion for selecting $\mathcal{H}^*$.

Reviewer comment (Experiments):
The paper does not do a good job at justifying the practical significance of handling unspecified forward operator, and demonstrating that the proposed problem setting is not contrived. None of the degraded images used in this paper is really from the truly real-world setting where the forward operator type is completely unknown. The demonstrated results still assume that the unspecified forward model is one of three known forward operator types.

Author response:
We recognize that our demonstrations, focusing primarily on unspecified known forward operator types, might not fully capture the complexities of real-world scenarios where the forward operator is completely unknown. This is a common challenge in the field, as even training-based, all-in-one restoration methods typically restore images with degradation types that are represented in their training set.
In our experiments, we deliberately chose degradations that represent typical inverse problems, aiming to provide a realistic yet controlled experiment environment. Our proposed method introduces reduced restrictions in problem settings, which enables more flexible and comprehensive solutions in complex scenarios. This approach marks a significant step toward addressing real-world settings more effectively.

Reviewer comment (Evaluation):
The paper does not provide any information on the computational cost of the proposed method and how it compares to the baselines.

Author response:
Thank you for your comment regarding the computational cost of our proposed method. To address this, we have included a detailed Efficiency Analysis in the appendix of our paper.

Reviewer comment (Novelty):
Because this paper only considers a finite number (really just three) of forward operator types, the problem seem to really boil down to just running multiple reconstructions in parallel, each assuming a different forward operator type.

Author response:
To clarify, the selection of a finite number of forward operator types in our experiments was primarily driven by the need for simplicity and clarity in demonstrating our method. This aligns with common practices in the field, where even training-based, all-in-one restoration methods typically focus on a limited range of degradation types represented in their training datasets.
Beyond this, our framework goes beyond simply executing parallel reconstructions. We incorporate an innovative approximation algorithm designed to progressively enhance the accuracy of latent variable estimates. In addition, we utilize Maximum a Posteriori (MAP) estimation for precise tuning both the forward operator and its associated parameters. The combination of these strategies distinctly sets our framework apart from standard parallel reconstruction approaches, offering enhanced reconstruction quality.

Reviewer comment (Experiments):
If one runs multiple separate reconstruction in parallel, each under a different forward operator assumption, how would this framework be different in terms of quality and computational cost?

Author response:
To clarify, the selection of a finite number of forward operator types in our experiments was primarily driven by the need for simplicity and clarity in demonstrating our method. This aligns with common practices in the field, where even training-based, all-in-one restoration methods typically focus on a limited range of degradation types represented in their training datasets.
Beyond this, our framework goes beyond simply executing parallel reconstructions. We incorporate an innovative approximation algorithm designed to progressively enhance the accuracy of latent variable estimates. In addition, we utilize Maximum a Posteriori (MAP) estimation for precise tuning both the forward operator and its associated parameters. The combination of these strategies distinctly sets our framework apart from standard parallel reconstruction approaches, offering enhanced reconstruction quality.

Reviewer comment (Experiments):
Are there any real-world examples (i.e., not controlled simulations on FFHQ) that can show exactly why this problem is meaningful?

Author response:
Thank you for your question regarding the real-world applicability of our research. The inverse problems we have addressed in our experiments, such as those presented in the FFHQ, AFHQ, and ImageNet datasets, are representative of common challenges in the field of inverse problems. The diversity and complexity of the data in these datasets provide a robust platform for demonstrating the efficacy of our approach in various contexts. We understand the importance of illustrating the practical significance of our work and are continually seeking opportunities to apply our method to real-world examples that further demonstrate its utility and impact.

Reviewer comment (Presentation):
As written, lines 7-12 of Algorithm 1 are rewriting the same values over and over.

Author response:
The variable $i$ represents the iteration round within each timestep. We recognize that this approach results in some repetition of equations within the algorithm box. However, this repetition was intentional to preserve the integrity and clarity of the algorithm's presentation. It ensures that each step is explicitly stated, avoiding any ambiguity about the process or the sequence of operations.

Reviewer comment (Theory):
The SSIM between x and y is not a good model for P(H|x,y,\phi).

Author response:
In our methodology, the diffusion model's output is the estimated image $\mathbf{x}_0$, and it is critical to recognize that this output cannot be directly compared with the measurement $\mathbf{y}$. To facilitate a meaningful comparison, we apply the forward function to the estimated image $\mathbf{x}_0$. This ensures that both the measurement $\mathbf{y}$ and the estimated image $\mathbf{x}_0$ are processed under comparable conditions, thus maintaining the integrity and relevance of the comparison.

Crucially, our proposed method is indeed capable of handling operations like inversion or flipping, which might cause $\mathbf{y}$ to appear dissimilar to $\mathbf{x}_0$. This is because our method does not rely on direct comparisons between $\mathbf{y}$ and  $\mathbf{x}_0$, but rather compares images after they have been subjected to the same forward function. This method effectively accounts for transformations that might otherwise complicate direct comparisons, ensuring accurate and reliable assessments regardless of such operations.

Reviewer comment (Novelty):
The paper doesn't motivate the problem. Without a real-world application to point to, the proposed problem setting seems somewhat contrived.

Author response:
We appreciate your comment regarding the motivation of the problem presented in our paper. Our research introduces a new, challenging setting that expands the scope of traditional inverse problems and blind inverse problems. While we recognize that a specific real-world application was not explicitly detailed, the broader aim of our work is to establish a foundational approach that can be adapted to a wide range of practical applications in the future.
The reduced restrictions in our proposed problem setting allow for more flexible and comprehensive solutions in complex scenarios where traditional methods may fall short. This has significant potential in fields such as medical imaging, signal processing, and other areas where accurate image reconstruction is crucial despite incomplete or altered data. Furthermore, our methodology's effectiveness and adaptability, particularly in the selection of forward measurement strategies, opens up avenues for extensive future research.

Reviewer comment (Theory):
I don't think the two sides of equation (10) are equivalent (or at least it's nonobvious why this would be the case).

Author response:
Equation (10) is based on the application of Bayes' Theorem in the context of Maximum A Posteriori (MAP) estimation. The theorem states that the posterior probability is proportional to the likelihood of the data given the parameters multiplied by the prior probability of the parameters. While the equivalence may not be immediately obvious, it is grounded in established probabilistic principles and the specific assumptions of our model.

Reviewer comment (Theory):
In most applications p(h) in equation (10) isn't constant.

Author response:
The prior of $\mathcal{H}$  is omitted since we model it as a constant. This assumption is based on our decision to model the prior distribution over possible operators as a simple uniform distribution. This simplification is made for reasons of computational efficiency and simplicity in the modeling process.

Reviewer comment (Reproducibility):
In lines 7-12 of Algorithm 1, should "i" be indexing something? H? As written, the same values are being overwritten over and over again.

Author response:
The variable $i$ represents the iteration round within each timestep. We recognize that this approach results in some repetition of equations within the algorithm box. However, this repetition was intentional to preserve the integrity and clarity of the algorithm's presentation. It ensures that each step is explicitly stated, avoiding any ambiguity about the process or the sequence of operations.

Reviewer comment (Theory):
"Here, we use the structural similarity index measure (SSIM) to approximate the p(H | x0, y,φ). We measure the similarity of the measurement image and the estimated image operated by the forward function to obtain the optimal H∗." Does this mean the proposed method can't handle simple operations (e..g, inversion or flipping) that cause y not to look like x?

Author response:
In our methodology, the diffusion model's output is the estimated image $\mathbf{x}_0$, and it is critical to recognize that this output cannot be directly compared with the measurement $\mathbf{y}$. To facilitate a meaningful comparison, we apply the forward function to the estimated image $\mathbf{x}_0$. This ensures that both the measurement $\mathbf{y}$ and the estimated image $\mathbf{x}_0$ are processed under comparable conditions, thus maintaining the integrity and relevance of the comparison.

Crucially, our proposed method is indeed capable of handling operations like inversion or flipping, which might cause $\mathbf{y}$ to appear dissimilar to $\mathbf{x}_0$. This is because our method does not rely on direct comparisons between $\mathbf{y}$ and  $\mathbf{x}_0$, but rather compares images after they have been subjected to the same forward function. This method effectively accounts for transformations that might otherwise complicate direct comparisons, ensuring accurate and reliable assessments regardless of such operations.

Reviewer comment (Writing):
Is big N in line 7 of algorithm 1 the same as little n in the definition of S_H?

Author response:
No, big $N$ in line 7 of Algorithm 1 and little $n$ in the definition of $\mathcal{S}_\mathcal{H}$ are not the same. 
In our algorithm, big $N$ specifically denotes the number of iterations within each timestep. 
This iteration count is crucial for updating both the image $\mathbf{x}_t$ and the parameter $\boldsymbol{\varphi}$. 
The iterative process allows for a synergistic improvement: a better-estimated parameter $\boldsymbol{\varphi}$ leads to a more accurate estimation of the image $\mathbf{x}_t$, and conversely, a more accurately estimated image enhances the estimation of the parameters.

Reviewer comment (Writing):
Pg 2: "Our UFODM, simultaneous estimation of both the restored image and the forward operator’s parameters"-->"Our UFODM, which simultaneoulsy estimates both the restored image and the forward operator’s parameters"

Author response:
Thank you for pointing out these typos and areas that require clarification in our manuscript. We will ensure these are corrected in our revised paper.

Reviewer comment (Writing):
Pg 4: "Naturally, this setup is more challenging than the traditional (blind) inverse problem, as it involves an unspecified forward operator and a significantly larger solution space." Should this read "(non-blind)"?

Author response:
Regarding your comment on Page 4: 'Naturally, this setup is more challenging than the traditional (blind) inverse problem,' we appreciate the opportunity to clarify. In this context, 'traditional' refers to non-blind inverse problems, where the forward function's parameters are known. In contrast, a blind inverse problem involves estimating both the image and the parameters of the forward function. Our method addresses a more challenging setup, involving an unspecified forward operator and a significantly larger solution space, thereby extending beyond the scope of traditional and blind inverse problems. This clarification will be added to ensure the distinction between these problem settings is clear.

Reviewer comment (Writing):
Pg 5: "the perturbed image of measurement y" "image of" seems unnecessary.

Author response:
Thank you for pointing out these typos and areas that require clarification in our manuscript. We will ensure these are corrected in our revised paper.