Paper: Prior Mismatch and Adaptation in PnP-ADMM with a Nonconvex Convergence Analysis (ICLR-2024)

Reviewer comment (Theory):
At first, I just wanted to state that the theorems are impossible: As $(x^0, z^0, s^0)$ is the (arbitrary) starting point of the ADMM algorithm, it is impossible to ensure that $\phi(x^0, z^0, s^0) - \phi^* \geq 0$, which immediately contradicts the inequality in Theorem 2. Yet, I think this is just a typo, i.e., one needs to start with the first iteration of the ADMM algorithm to make the theorem work. I'd like to ask the authors to briefly confirm this as I have not checked any proof.

Author response:
Thank you for pointing out this inconsistency in our notations. We have revisited the statements for our assumptions, theorems, and proofs to streamline the presentation. The updated theorems statement combines all the constants into $A_1$ and $A_2$, while the proof explicitly shows that $ϕ(x^0,z^0,s^0)−ϕ^∗\geq0$, by using Lemma 3 that shows that the augmented Lagrangian monotonically decreases for the iterations of PnP-ADMM.

Reviewer comment (Theory):
The practical implications of the paper are not large (= it is good to know that a setting similar to what everyone does in practice converges, but the assumptions are of course never verifiable exactly; knowing a constant by which the gradient of the objective differs from zero still makes it hard to say something about the point one converges to).

Author response:
Indeed, we revised the discussion after our theorems to make the statements more precise.

Reviewer comment (Theory):
Even after realizing that the theorems' starting points might just have been a typo, the results still appeared to be a little surprising (as the augmented Lagrangian is not monotone in the usual convex ADMM setting). Then, I, however, remembered the paper "Global Convergence of ADMM in Nonconvex Nonsmooth Optimization", where the augmented Lagrangian was also used as a Lyapunov function (which might therefore be worth citing). How similar/different is the conducted convergence analysis from taking existing literature on the analysis of nonconvex ADMM (the one I cited, versions that make smoothness assumptions, and those that have possibly considered an inaccurate evaluation of the prox) and just making assumptions that allow you to return to such a known analysis? Things like 'summable errors' in substeps of such algorithms are also very common in the general optimization literature.

Author response:
Thank you for providing [1], which we have cited in the revised manuscript. The significance of our analysis lies in integrating the concepts and assumptions from PnP to obtain a novel analysis of corresponding ADMM iterations. The traditional literature on ADMM does not leverage the specific properties of PnP operators, while the current PnP literature has not considered a nonconvex analysis of PnP-ADMM under mismatched deep denoisers. This allows us to bring unique insights on a popular topic within the imaging inverse problems community.

[1] Wang, Yu, Wotao Yin, and Jinshan Zeng. "Global convergence of ADMM in nonconvex nonsmooth optimization." Journal of Scientific Computing 78 (2019): 29-63.

Reviewer comment (Theory):
In assumption 6, one needs the inequality to be satisfied for all stationary points of the augmented Lagrangian, and the theorems are to be interpreted as "there exists $(x^*, z^*, s^*)$ such that ...", correct? Does the existence of a stationary point have to be assumed or do the assumptions guarantee that?

Author response:
We appreciate the reviewer's feedback. We have updated the manuscript to streamline the presentation of our results and show the existence of the infimum of the augmented Lagrangian.

Reviewer comment (Theory):
Can you please comment on assumption 1: Isn't it common to assume that "realistic" images lie in some set (e.g. a union of subspaces) much smaller than the dimension of the space? Would that cause a problem for assumption 1?

Author response:
While the “union of subspaces” model is not compatible with our Assumption 1, there are many other “sparse” models that are. Any distribution that has density everywhere, even when that density has very low probability works. For example, Assumption 1 is compatible with the assumption that image coefficients are distributed according to the Bernoulli-Gaussian distribution (i.e., either zero or Gaussian coefficients). It is also worth noting that Assumption 1 can be relaxed to generalize the theory to arbitrary priors (see Remark 2 in (Gribonval, 2011)). The benefit of the current presentation is the simplicity, which doesn’t really reduce generality.

Reviewer comment (Experiments):
The paper's experiments primarily focus on super-resolution and deblurring with negligible measurement noise, limiting the comprehensiveness of the investigations. It lacks exploration of scenarios with nonconvex data-fidelity terms, which would enrich the analysis and diversify the paper's empirical scope.

Author response:
Prompted by your comment, we have included the results for the phase retrieval problem, which is nonconvex, in Section H.3 of the supplementary materials.

Reviewer comment (Novelty):
The related work presented in the paper lacks depth and breadth (e.g., in domain adaptation, and review of previous work).

Author response:
We have added a paragraph to the Background Section in Page 3 of the main paper, addressing domain adaptation in imaging inverse problems.

Reviewer comment (Theory):
A table of assumptions for the cited references regarding the theoretical convergence analyses of PnP should be considered. Such a table would be valuable for readers to understand the differences between these works and the current study.

Author response:
Prompted by your comment, we have added Table 3 to Section G of the Supplementary materials for a comparison of existing theoretical analyses of PnP methods.

Reviewer comment (Experiments):
The experiments detailed in Section 4 raise a query: is the measurement noise level consistently set at 0.01? If so, this level of noise appears comparable to the noise-free scenario. It would significantly enhance the paper to explore the impact of more robust and varying levels of measurement noise on the obtained results, underlining the model's performance in noisier conditions.

Author response:
Prompted by the feedback, we have added Tables 4 and 5 to Section H.1 of the Supplementary material, presenting super-resolution results obtained with a measurement noise set to 0.03.

Reviewer comment (Experiments):
While the experiments concentrate on convex data fidelity terms using the $\ell_2$ norm, it would be advantageous to broaden the scope by including experiments with nonconvex data-fidelities. Extending the experiments to encompass nonconvex scenarios would further corroborate the claims made in the paper.

Author response:
Prompted by feedback, we have included the results for a new phase retrieval experiment with a nonconvex data-fidelity term, in Section H.3 of the Supplementary materials.

Reviewer comment (Writing):
The paper presents Theorems 1 and 2, providing theoretical support for the proposed methodologies. It would be beneficial to provide a descriptive narrative of the results or outcomes derived from these theorems to facilitate a more comprehensive understanding for readers.

Author response:
We have adjusted the last two paragraphs of Section 3.1 to better clarify our results.

Reviewer comment (Writing):
Sections A-D of the Appendices delve into the topic of “the optimality condition for mismatched MMSE denoiser.” Further explanation is required to enhance the clarity and comprehension of this topic.

Author response:
We have added additional clarification to the revised manuscript.

Reviewer comment (Theory):
An observation on the gradient of h in Eq.(16) leads to a query: shouldn't the gradient of Eq.(18) be zero? As $\nabla \frac{1}{2\gamma}\|z-(x^k+s^{k-1})\|_2^2 = \frac{1}{\gamma}(z-(x^k+s^{k-1}))$ and $\nabla h(z) = \frac{1}{\gamma} (x^k+s^{k-1}-z)$.

Author response:
The gradient of Equation (18) is zero only for $ \overline{z}^k$, a fact utilized in the subsequent equations on the same page, specifically the one preceding Equation (19).

Reviewer comment (Evaluation):
What if the there is no ground truth in a real-world test dataset? Does it mean that we have to manually label find several images which are close to the test set?

Author response:
**No paired data.** Both theoretical and experimental results suggest that for closer data distributions, we get lower reconstruction errors. One could thus use a dataset closest to the inference dataset to fine-tune the pre-trained denoisers to achieve lower reconstruction error.