Paper: Self-supervision Meets Bootstrap Estimation: New Paradigm for Unsupervised Reconstruction with Uncertainty Quantification (ICLR-2024)

Reviewer comment (Experiments):
It is highly expected that the paper includes a comparison with other typical methods for MRI reconstruction, such as [1] and [2]. Although the authors argue that deep learning models may suffer from unreliability, as the fast growing of vision transformers, their appealing performance should not be ignored. Thus, a comparison with such methods will make the results more convincing. Although these methods may not focus on zero-shot, the testing performance can still be compared.

Author response:
While we would love to add more comparison with other advanced MRI reconstruction methods, unfortunately, it requires substantial time and access to their original codes (some are not shared though) for an informed and detailed comparison. We add citations to these referred papers, and will further explore them in our future work.

Reviewer comment (Theory):
I think the paper lacks a discussion of the differentiability of the aggregation function h. Although the classical MSE loss is differentiable, due to the use of the aggregation function in the MSE which is adopted to train a model, the authors are highly encouraged to discuss the differentiability of the aggregation function.

Author response:
The differentiability of the aggregation function becomes a main concern in our derivation but we found it operable for  backpropagation and thus it won’t influence the optimization. For example:
   - With or without the aggregation function, the gradients of the reconstruction model parameterized by $\theta$ are always: 
$$\frac{\partial{\mathcal{L}}}{\partial{\theta}}=\frac{\partial{\mathcal{L}}}{\partial{f_{AF}}}\frac{\partial{f_{AF}}}{\partial{f}}\frac{\partial{f}}{\partial{\theta}}=\frac{\partial{\mathcal{L}}}{\partial{f}}\frac{\partial{f}}{\partial{\theta}}$$
   - As for the gradients of $y$ and $U$, we add an explanation after Equation 7, that aggregation function results in a re-undersampling mask which may not influence the differentiability to $y$. The gradients of $U$, on the other hand, may be influenced, but they are not optimized for MRI reconstruction, thus it may not a problem.

Reviewer comment (Evaluation):
Is the uncertainty in this paper partially from the resampling operation due to the use of bootstrap? More insights will be helpful.

Author response:
In the original version of our paper, uncertainty has been limited to the randomness of undersampling, while the Bootstrap resampling actually “simulates” it. It’s true that for the reconstruction of re-undersampled images, Bootstrapping incorporates uncertainty, but in inference, no re-undersampling will be conducted.

Reviewer comment (Experiments):
Although reconstruction is an important topic in MRI, are there any other reasons that make the proposed method tie to MRI reconstruction? In other words, whether the proposed method is suitable for potential reconstruction tasks in natural image domains?

Author response:
We’re extending our work as we discussed in Conclusion. There would be a trivial extension that applies virtual sample and pseudo resampling in similar tasks related to Compressed Sensing. Other image-to-image translation tasks like super-resolution and deblurring could also benefit from our work by defining virtual sample sets in particular domains. We’re open to further comments.

Reviewer comment (Experiments):
Is that possible to use the Monte Carlo method in this work?

Author response:
At this stage, Mote Carlo method would not be appliable as it incorporates sequential sampling and requires extra control for the  re-undersampling purpose, etc. We will further explore this idea.

Reviewer comment (Experiments):
I think one of my biggest concerns is how the proposed method compared with other existing approaches for uncertainty quantification, there have been a wide range of works on this topic, they are either sampling based [Uncertainty Quantification in Deep MRI Reconstruction] or directly estimation the absolute residual error [Rigorous Uncertainty Estimation for MRI Reconstruction], this paper lack the comparisons with other approaches, please discuss/cite them.

Author response:
**The evaluation of UQ**
- We add more discussion on UQ in the section on Related Work while focusing our quantification on MSE estimation in UQ. We also discuss the work you referred to. The UQ in our work is specifically referred to error estimation.
- We compare the estimation of MSE with residual magnitude regression of the recommended work [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging] (which should cover the family of similar methods) and SURE estimation of [Uncertainty Quantification in Deep MRI Reconstruction] in Appendix F. Unfortunately we cannot compare our work with the method in [Rigorous Uncertainty Estimation for MRI Reconstruction] because we do not have access to that paper's materials. We are investigating the implementation of more methods and further analysis may also be added later.
- For evaluation metrics, we find the correlation between the estimated and actual MSE serving as a good indicator for MSE-based UQ, since the ground truth is clear. Other existing evaluation measures mainly focus on variance and quantiles, so we are not able to use them.

Reviewer comment (Evaluation):
For the reconstruction results, the authors only showed an PSNR and SSIM plot (Figure 7) without any visual results to inspect on the details, the only visual results is Figure 5, which also doesn't deliver much information. I think this paper demonstrates a proof-of-concept, but lack evaluations.

Author response:
**Visual Presentation of Reconstruction Results**
- Visual examples including the reconstruction results and error maps are added to Figure 8 in the revised paper.

Reviewer comment (Evaluation):
What is the purpose of estimating MSE, this can be generalized to an open question, how to use the uncertainty estimation results for diagnosis, I can imagine it would be useful if we compute uncertainty in latent space, but could you elaborate on how to use uncertainty estimation for down-stream task? One inspiring paper of quantifying theoretical results of uncertainty estimation is: [https://arxiv.org/abs/2202.05265 Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging], you maybe able to get some insights from.

Author response:
**The application of UQ**
- Specifically for our work, the purpose of estimating MSE has a rather simple and special answer: that is to optimize it. That’s the core idea of our proposed self-supervised MRI reconstruction method, please refer to Section 5.3. Please be noted, this may not be generalized to other UQ notions or methods since they may either need training itself or be not suitable for optimization.
- The application of general UQ is worthy of discussion. We add a brief discussion at the beginning of Section 2.3a.

Reviewer comment (Writing):
Excessive Equations and Notations: The paper is overwhelmed with equations and notations, overshadowing the fundamental concept. The core idea appears to be sampling, but the multitude of equations adds unnecessary complexity without aiding comprehension.

Author response:
We add a note to the Introduction section directing readers to the table of notations in the appendix. The equations in the main content are largely reduced as well (from 25 to 16).

Reviewer comment (Writing):
Confusing Notations: Notations like the two 'U's in Equation (6) are ambiguous and visually similar, leading to confusion and hindering understanding.

Author response:
Do you mean the Us in Equation(5)? The two $$U$s refer to the “random variable” and “vector value” of the mask $U$ with display styles provided by the ICLR formatting instructions. Anyway, we remove the equations.

Reviewer comment (Writing):
Lack of Explaining Prior Works: Assumptions about the reader's familiarity with existing work, especially (Yaman, 2022)'s zero-slot learning, create gaps in understanding. The absence of pertinent details hampers comprehension.

Author response:
We add a new figure - Figure 1 - to summarize the framework of MRI reconstruction, which may help explain our work better. The detailed settings of the previous methods are provided in Appendix F.

Reviewer comment (Evaluation):
Unclear Significance of Variance: The paper lacks clarification on why the variance (uncertainty) of bootstrap resampled reconstruction is important. Address the relevance of this aspect, especially in comparison to uncertainty quantification for raw measurement reconstruction.

Author response:
We agree and notice the inclusion of variance estimation becomes redundant and compromises the integrity of the article. Such discussion may be pertinent to another independent research.