Paper: Monte Carlo guided Denoising Diffusion models for Bayesian linear inverse problems. (ICLR-2024)

Reviewer comment (Novelty):
It seems the contribution point 3 is mainly a support for the first two points, hardly it can be classified as an independent contribution.

Author response:
We think that the point 3 is an empirical contribution in itself since we show that the current methods used for solving linear inverse problems with diffusion do not actually correctly sample the posterior in the most straightforward cases.

Reviewer comment (Experiments):
This method should be pretty slow as both diffusion model and particle filter are computationally heavy ones. Would be good to clearly state the limitation and also report the runtime etc.

Author response:
While we agree that our method is computationally more expensive than existing works such as DDRM and DPS, we believe that this is the price to pay to obtain more faithful approximations of the posterior. Indeed, as we show in the Gaussian mixture experiment, DPS and DDRM do not actually sample approximately from the posterior and cannot be made more precise using more computational resources. On the other hand, our method has an extra parameter (the number of particles $N$) which controls the accuracy of the approximation. We firmly believe that there is an intrinsic tradeoff between accuracy and computational burden when dealing with the posteriors of generative models. That being said, in moderate dimensions, drawing $N$ parallel samples from DPS or DDRM has the same computational cost as running one particle filter with $N$ particles, which returns $N$ samples that approximate the posterior (the resampling cost is negligible in front of the cost of running the diffusion). In much larger dimensions, taking $N$ particles in MCGDiff will result in $N$ collapsed particles, which is not qualitatively the same thing as running DPS or DDRM with $N$ samples. However, we advocate in our experiments that in such cases, one should use a small number of particles and instead run $M$ parallel MCGDiff with $N/M$ particles, to obtain a particle approximation of the posterior with $M$ different and independent particles. In such cases, the computational cost of our algorithm is larger than that of DPS and DDRM (in the sense that with the same computational cost you can draw $N$ i.i.d. samples from DPS and DDRM). While it is true that the particles from a single particle filter lack diversity in global level features, still, we believe that the quality of our proposal allows us to draw samples in the support of the posterior, and not outside, as it is the case of the other methods. Hence running parallel particle filters allows us to have samples that are globally diverse.

Reviewer comment (Evaluation):
It seems very difficult to conclude that the proposed method outperforms the others in image dataset, is there any other ways to quantitatively demonstrate the advantage of the proposed method?

Author response:
For our imaging experiments, we believe that no method can quantitatively assert the performances of all three algorithms, as we do not have samples from the posterior to compare to. This is precisely the reason why we focus on tractable experiments in the main paper. In the appendix though, we have compared MCGDiff and other algorithms in some image tasks by at first generating 100 samples from the posterior and then showing the 2 samples from those that are the furthest apart. We can see that MCGDiff seems to obtain more coherent samples, even though, as we said before, this is purely a qualitative evaluation and is of course subject to discussion.

Reviewer comment (Reproducibility):
The code issues have not been addressed. From looking at the source, the diffusion code seems to be based on DDPM, which is not acknowledged in the main text or appendices. Speed issues have not been mentioned.

Author response:
Dear reviewer,  
We thank you for the time and effort taken to review our manuscript and the positive feedback. We thank you for the points that you raised concerning the bibliography and the speed issues and will be integrating it into the main text in the future. We will also add a precise pseudo code that reflects exactly the source code, in the appendix.

Reviewer comment (Novelty):
The bibliography is excellent except for the first paragraph of the introduction. Linear inverse problems as described have been studied mathematically for a very long time (Fredholm, etc.) and in computer vision since the late 1980s at least. The bibliography should perhaps mention this.

Author response:
Dear reviewer,  
We thank you for the time and effort taken to review our manuscript and the positive feedback. We thank you for the points that you raised concerning the bibliography and the speed issues and will be integrating it into the main text in the future. We will also add a precise pseudo code that reflects exactly the source code, in the appendix.

Reviewer comment (Evaluation):
The experiments on GM seem to indicate that MCGdiff does sample from the posterior distribution but not necessarily in a "thorough manner" (see Fig. 5 and 6). How can this be better handled?

Author response:
Indeed in figure 5 and 6 we see that unlike the other existing methods, MCGDiff finds all the modes but in some cases the tails of each mode are thinner. A straightforward answer to your question, relying on the theorems presented in the main paper, would be to increase the number of particles being used. There are also other alternatives, as for example using the unadjusted Langevin algorithm initialized at the particles resulting from MCGDiff and targeting the posterior of the model. This requires having the score at time 0, to which we may not have access as the data distribution may not have a density in practice. However, we can replace it with the score at some timestep epsilon, close to 0. We have tried this method and in some cases it leads to better spreading around the mode and smaller sliced Wasserstein distance.

Reviewer comment (Experiments):
How to deal with noise present in the training dataset?

Author response:
The current work is an effort to handle bayesian linear inverse problems when using a pre-trained denoising diffusion generative model as a prior, without any need of (further) training. In this sense, the measurement $y$ can be a noisy linear observation of the state. But if we understand your question correctly, it concerns the training data for the underlying generative model, which we do not have control over as we use off the shelf trained diffusion models. However, if one has some prior information on the noise present in the dataset, perhaps there might a path forward. This is a really interesting yet difficult question that we believe is outside the scope of the current paper.

Reviewer comment (Experiments):
It would have been intriguing to explore the performance of the MCGDiff algorithm on non-image data. While the research focuses on image-related tasks, extending the investigation to other data types would provide a broader perspective on the algorithm's applicability and effectiveness across various domains.

Author response:
Dear reviewer,
We thank you for the time and effort taken to review our manuscript. We are glad that you have appreciated our work and thank you for the positive feedback!

 The code that is currently available in the anonymous git will be available once the submission process is over. Indeed, we also believe that MCGDiff can have a broader impact beyond image-related tasks. We are currently using it on healthy electrocardiogram (ECG) to develop interpretable outlier detection tools for electrophysiological anomalies, with great success.

Reviewer comment (Experiments):
Have you explored the application of the MCGDiff algorithm in addressing inverse problems beyond image-related tasks, and if so, could you share any insights into its performance and adaptability in those contexts?

Author response:
Dear reviewer,
We thank you for the time and effort taken to review our manuscript. We are glad that you have appreciated our work and thank you for the positive feedback!

 The code that is currently available in the anonymous git will be available once the submission process is over. Indeed, we also believe that MCGDiff can have a broader impact beyond image-related tasks. We are currently using it on healthy electrocardiogram (ECG) to develop interpretable outlier detection tools for electrophysiological anomalies, with great success.