Paper: On the Posterior Distribution in Denoising: Application to Uncertainty Quantification (ICLR-2024)

Reviewer comment (Writing):
$\sigma$^2 not defined in Eq. 4.

Author response:
**“$\\sigma$^2 not defined in Eq. 4.”**

The standard deviation of the noise, denoted by $\\sigma$, is defined for Eq. (1), where the denoising problem is first introduced. The notation remains the same across the entire paper.

Reviewer comment (Novelty):
This particular method for image denoising uncertainty quantification is not motivated. Why would one use this approach? Why not just write down the posterior distribution using a likelihood model and the pretrained score-based model as (log) prior and use MCMC? Why not use amortized Bayesian inference and just simply sample from the posterior distribution?

Author response:
**“This particular method for image denoising uncertainty quantification is not motivated. Why would one use this approach? Why not just write down the posterior distribution using a likelihood model and the pretrained score-based model as (log) prior and use MCMC? Why not use amortized Bayesian inference and just simply sample from the posterior distribution?
Related to the previous point, a lot of emphasize has been given to higher-order moments of the posterior. Why do we care about computing these? Can't we compute these quantities using samples from the posterior distribution?”**

First, note that a denoiser pretrained for a specific noise level (as we use in this work) cannot be used to generate posterior samples using MCMC. This is because the denoiser corresponds to the score of the noisy image distribution, not the clean one (namely it relates to $\\log p_\\mathbf{y}$ and not to $\\log p_\\mathbf {x}$). Approaching $\\log p_\\mathbf {x}$ is possible with a denoiser that is trained conditioned on the noise level and can handle a large range of noise-levels (e.g., as in denoising score matching and in diffusion models). Our method, on the other hand, can use *any* black-box denoiser, architecture-agnostically, that can handle only the noise-level of the test images we wish to analyze.

Now, it is certainly possible to resort to some method for generating posterior samples, as you suggest. But note that simply sampling from the posterior does not provide the user with quantitative tools for assessing uncertainty, and it may also require the user tedious scrolling through thousands of samples to try to visually grasp the likelihood of each possible semantic variation in the output. Yet, as you correctly note, given many samples from the posterior, it is possible to compute e.g. the principal components (PCs) of the posterior.

We followed your suggestion and compared our method to this baseline (see App. K in the new manuscript). Specifically, we used DDNM [Wang et al., 2023] to generate many samples from the posterior and then performed PCA on those samples to extract the posterior PCs. As can be seen in Fig. S4, as the number of posterior samples increases, the PCs estimated using this baseline become cleaner and more similar to our PCs. However, note that even with 500 samples, the PCs of this baseline do not seem to have fully converged, and generating each sample requires 100 neural function evaluations (NFEs) with DDNM, so that generating 500 posterior samples requires 50,000 NFEs (just for analyzing the uncertainty for a single noisy input image!). In contrast, our method can faithfully extract each posterior PC with only 10 NFEs (see convergence graph in Fig. A1). Therefore, for extracting e.g. 5 PCs, our method is roughly 1000x faster than that naïve approach.

We also added a numerical comparison to that baseline. Due to the heavy computations involved in generating posterior samples with DDNM, for now we conducted comparisons only with the 4 noisy images of Fig. 2, but we will include the results for a large dataset in the revised manuscript. The comparison is as follows. First, the empirical mean of the samples generated by DDNM should theoretically approximate the posterior mean, which is the MSE-optimal restoration. As we verify, this estimate is indeed very close to our denoiser’s output, and they both achieve practically the same MSE to the ground-truth images. Second, we computed the PCs from the posterior samples (after subtracting their mean). We compared those PCs to the PCs of our method by measuring the norm of the error after projecting it onto these PCs. The larger this norm, the larger the portion of the error that these PCs account for (larger is better). Mathematically, this measure is defined as $\\|\\boldsymbol{V}^T(\\boldsymbol{x} - \\mu\_1(\\boldsymbol{y}))\\|^2\_2$, where $\\boldsymbol{V}$ is a matrix containing the PCs as columns. We found that the mean of this measure for both methods is very close and ours is even slightly better: ours is 0.81 and the baseline’s is 0.63.

Thank you for this suggestion.

Reviewer comment (Theory):
Related to the previous point, a lot of emphasize has been given to higher-order moments of the posterior. Why do we care about computing these? Can't we compute these quantities using samples from the posterior distribution?

Author response:
**“This particular method for image denoising uncertainty quantification is not motivated. Why would one use this approach? Why not just write down the posterior distribution using a likelihood model and the pretrained score-based model as (log) prior and use MCMC? Why not use amortized Bayesian inference and just simply sample from the posterior distribution?
Related to the previous point, a lot of emphasize has been given to higher-order moments of the posterior. Why do we care about computing these? Can't we compute these quantities using samples from the posterior distribution?”**

First, note that a denoiser pretrained for a specific noise level (as we use in this work) cannot be used to generate posterior samples using MCMC. This is because the denoiser corresponds to the score of the noisy image distribution, not the clean one (namely it relates to $\\log p_\\mathbf{y}$ and not to $\\log p_\\mathbf {x}$). Approaching $\\log p_\\mathbf {x}$ is possible with a denoiser that is trained conditioned on the noise level and can handle a large range of noise-levels (e.g., as in denoising score matching and in diffusion models). Our method, on the other hand, can use *any* black-box denoiser, architecture-agnostically, that can handle only the noise-level of the test images we wish to analyze.

Now, it is certainly possible to resort to some method for generating posterior samples, as you suggest. But note that simply sampling from the posterior does not provide the user with quantitative tools for assessing uncertainty, and it may also require the user tedious scrolling through thousands of samples to try to visually grasp the likelihood of each possible semantic variation in the output. Yet, as you correctly note, given many samples from the posterior, it is possible to compute e.g. the principal components (PCs) of the posterior.

We followed your suggestion and compared our method to this baseline (see App. K in the new manuscript). Specifically, we used DDNM [Wang et al., 2023] to generate many samples from the posterior and then performed PCA on those samples to extract the posterior PCs. As can be seen in Fig. S4, as the number of posterior samples increases, the PCs estimated using this baseline become cleaner and more similar to our PCs. However, note that even with 500 samples, the PCs of this baseline do not seem to have fully converged, and generating each sample requires 100 neural function evaluations (NFEs) with DDNM, so that generating 500 posterior samples requires 50,000 NFEs (just for analyzing the uncertainty for a single noisy input image!). In contrast, our method can faithfully extract each posterior PC with only 10 NFEs (see convergence graph in Fig. A1). Therefore, for extracting e.g. 5 PCs, our method is roughly 1000x faster than that naïve approach.

We also added a numerical comparison to that baseline. Due to the heavy computations involved in generating posterior samples with DDNM, for now we conducted comparisons only with the 4 noisy images of Fig. 2, but we will include the results for a large dataset in the revised manuscript. The comparison is as follows. First, the empirical mean of the samples generated by DDNM should theoretically approximate the posterior mean, which is the MSE-optimal restoration. As we verify, this estimate is indeed very close to our denoiser’s output, and they both achieve practically the same MSE to the ground-truth images. Second, we computed the PCs from the posterior samples (after subtracting their mean). We compared those PCs to the PCs of our method by measuring the norm of the error after projecting it onto these PCs. The larger this norm, the larger the portion of the error that these PCs account for (larger is better). Mathematically, this measure is defined as $\\|\\boldsymbol{V}^T(\\boldsymbol{x} - \\mu\_1(\\boldsymbol{y}))\\|^2\_2$, where $\\boldsymbol{V}$ is a matrix containing the PCs as columns. We found that the mean of this measure for both methods is very close and ours is even slightly better: ours is 0.81 and the baseline’s is 0.63.

Thank you for this suggestion.

Reviewer comment (Theory):
How would one use this method when the noise level is unknown?

Author response:
**“How would one use this method when the noise level is unknown?”**

This is an interesting question, which we discuss in App. I.

While our theory holds only for the non-blind Gaussian denoising case, we empirically show in the paper that our approach provides sensible results also for the blind-denoising setting, in our experiment on real microscopy images (the bottom of Fig. 5), where the noise model is not known. There, the noise distribution in each pixel is likely closer to Poisson-Gaussian than to Gaussian, the noise level is unknown, and it is not even clear if the noise is completely white. To handle this task, we estimated $\\sigma$ using the naïve formula provided at the end of Sec. 3.3, and it can be seen that even in those cases our moment calculations, combined with our novel posterior PCA approach, seem to reveal meaningful uncertainty.

The subject of noise-level estimation, especially for AWGN, has been heavily researched, and state-of-the-art methods reach very-high precision [Chen et al., 2015; Khmag et al., 2018; Kokil & Pratap, 2021; Liu & Lin, 2012; Liu et al. 2013]. For example, when the real $\\sigma$ equals 10, the error in estimating sigma is around 0.05 (see e.g. [Chen et al., 2015]). Even our naïve estimation method mentioned above resulted in quite accurate results, but it is possible to use more advanced noise-level estimation methods to obtain a more accurate estimate for $\\sigma$ and then apply our method.

As we show in the appendix, the impact of small inaccuracies in $\\sigma$ on our uncertainty estimation is quite small. Fig. S3 shows that even for large errors of 5% in $\\sigma$ (which are significantly higher than the typical 0.5% errors for good noise level estimation techniques), our predicted eigenvectors are quite similar.

Our method can in fact be motivated for blind Gaussian denoising. Specifically, suppose that $\\mathbf{y}=\\mathbf{x}+\\sigma \\mathbf{n}$, where $\\mathbf{n}\\sim\\mathcal{N}(0,\\mathbf{I})$ is independent of $\\mathbf{x}$ and the noise level $\\sigma$ is a random variable. That is, the noise level is unknown to the denoiser (only the distribution of noise levels $p_\\sigma$ is known). In this case, it is easy to show using the law of total expectation that the Jacobian of the MSE-optimal denoiser is given by $\\mathbb{E}\left[\\frac{\textnormal{Cov}(\\mathbf{x}|\\mathbf{y},\\sigma^2)}{\\sigma}\middle|\\mathbf{y}=\\boldsymbol{y}\right]$. Namely, instead of $\\frac{\textnormal{Cov}(\\mathbf{x}|\\boldsymbol{y})}{\\sigma^2}$, which we had in the Gaussian setting, here the Jacobian reveals the *mean* of the posterior covariance divided by $\\sigma^2$. This matrix is a linear combination of the posterior covariances corresponding to different noise levels, so that it captures some notion of spread about the posterior mean, similarly to the regular posterior covariance that arises in the non-blind setting. Thus, intuitively, we expect that the top eigenvectors of this matrix capture meaningful uncertainty directions, similarly to the non-blind setting.

We added this intuitive explanation in the revised supplemental material, thanks.

Reviewer comment (Experiments):
This paper only considers AWGN removal, making its practical relevance to real-world denoising (e.g., Poisson, Laplacian, Speckle, etc) very limited. Moreover, it would be more interesting to show how this method can be applied to other imaging inverse problems like inpainting and super-resolution both empirically and theoretically.

Author response:
**“This paper only considers AWGN removal, making its practical relevance to real-world denoising (e.g., Poisson, Laplacian, Speckle, etc) very limited. Moreover, it would be more interesting to show how this method can be applied to other imaging inverse problems like inpainting and super-resolution both empirically and theoretically.”**

Thanks for this important point. First, we would like to comment that while the theory we developed indeed holds only for the non-blind Gaussian denoising case, we empirically show in the paper that our approach provides sensible results also in more generic denoising settings. For example, the bottom part of Fig. 5 illustrates the use of our approach on real microscopy images, in which the noise model is not known. In this setting, the noise distribution in each pixel is likely close to Poisson-Gaussian rather than Gaussian, the noise level is unknown, and it is not even clear if the noise is completely white. As can be seen, even in those cases our moment calculations, combined with our novel posterior PC approach, seem to reveal meaningful uncertainty.

Second, we would like to point out that our method can be motivated for slightly more general settings than just nonblind Gaussian denoising. In particular, it also makes a lot of sense for *blind* Gaussian denoising. Specifically, suppose that $\\mathbf{y}=\\mathbf{x}+\\sigma \\mathbf{n}$, where $\\mathbf{n}\\sim\\mathcal{N}(0,\\mathbf{I})$ is independent of $\\mathbf{x}$ and the noise level $\\sigma$ is a random variable. That is, the noise level is unknown to the denoiser (only the distribution of noise levels $p_\\sigma$ is known). In this case, it is easy to show using the law of total expectation that the Jacobian of the MSE-optimal denoiser is given by $\\mathbb{E}\left[\\frac{\textnormal{Cov}(\\mathbf{x}|\\mathbf{y},\\sigma^2)}{\\sigma}\middle|\\mathbf{y}=\\boldsymbol{y}\right]$. Namely, instead of $\\frac{\textnormal{Cov}(\\mathbf{x}|\\boldsymbol{y})}{\\sigma^2}$, which we had in the nonblind setting, here the Jacobian reveals the *mean* of the posterior covariance divided by $\\sigma^2$, where the mean is taken over all possible noise levels $\\sigma$. This matrix is a linear combination of the posterior covariances corresponding to different noise levels, so that it captures some notion of spread about the posterior mean, similarly to the regular posterior covariance that arises in the non-blind setting. Thus, intuitively, we expect that the top eigenvectors of this matrix capture meaningful uncertainty directions, similarly to the non-blind setting. We added this explanation in the revised supplemental material.

For general inverse problems, it is unlikely that there exists a closed-form expression allowing to extract the posterior covariance from a pre-trained MSE-optimal model. Perhaps there are convenient expressions for certain specific settings, but we leave this to future work. One option to address general inverse problems is via training-based solutions, however here we wanted to focus on revealing the information that is encoded within the weights of a *pre-trained* model. Another approach could be to use diffusion-based posterior sampling techniques (e.g. DDNM[Wang et al., 2023]) to generate many samples from the posterior and then perform PCA on those samples to extract the posterior PCs. However, note that the computational cost of such a technique would be very high, as each sample requires many neural function evaluations (NFEs) to generate (e.g. 100 in DDNM) and at least 500 samples are needed for obtaining accurate PCs (see the illustration we added in App. K). In contrast, our method can faithfully extract each posterior PC with only 10 NFEs (see convergence graph in Fig. A1). Therefore, for extracting e.g. 5 PCs, our method is roughly 1000x faster than that naïve approach. We added this discussion to App. J.

Reviewer comment (Experiments):
In this paper, no baseline methods are compared against with. No quantitative results about the uncertainty calibration such as expected calibration error etc. are reported.

Author response:
**“In this paper, no baseline methods are compared against with. No quantitative results about the uncertainty calibration such as expected calibration error etc. are reported.”, “Absolute error to the ground-truth should be also presented in Fig. 4 and Fig.5 to show that the uncertainty indeed can better reflect the restoration error.”**

Thanks for this important point. Please note that it is impossible to *directly* measure the quality of the estimated posterior PCs because denoising datasets contain only one clean image $\\boldsymbol{x}$ for each noisy image $\\boldsymbol{y}$. This single $\\boldsymbol{x}$ is just one sample from the posterior $p\_{\\mathbf{x}|\\mathbf{y}}$ and therefore it cannot be used to extract a “ground-truth” posterior covariance matrix or “ground-truth” PCs.

However, it is possible to compare our approach to a very strong baseline - that of diffusion-based posterior samplers, as mentioned above. We added this comparison to App. K. For each noisy image, we generated many posterior samples using DDNM[Wang et al., 2023] and used them to calculate the PCs of the posterior. As can be seen in Fig. S4, as the number of posterior samples increases, the PCs estimated using this baseline become cleaner and more similar to our PCs. However, note that even with 500 samples, the PCs of this baseline do not seem to have fully converged, and generating 500 posterior samples using DDNM requires 50,000 NFEs (just for analyzing the uncertainty for a single noisy input image!).

We also added a numerical comparison. Due to the heavy computations involved in generating posterior samples with DDNM, for now we conducted comparisons only with the 4 noisy images of Fig. 2, but we will include the results for a large dataset in the revised manuscript. The comparison is as follows. First, the empirical mean of the samples generated by DDNM should theoretically approximate the posterior mean, which is the MSE-optimal restoration. As we verify, this estimate is indeed very close to our denoiser’s output, and they both achieve practically the same MSE to the ground-truth images. Second, we computed the PCs from the posterior samples (after subtracting their mean). We compared those PCs to the PCs of our method by measuring the norm of the error after projecting it onto these PCs. The larger this norm, the larger the portion of the error that these PCs account for (larger is better). Mathematically, this measure is defined as $\\|\\boldsymbol{V}^T(\\boldsymbol{x} - \\mu\_1(\\boldsymbol{y}))\\|^2\_2$, where $\\boldsymbol{V}$ is a matrix containing the PCs as columns. We found that the mean of this measure for both methods is very close and ours is even slightly better: ours is 0.81 and the baseline’s is 0.63.

Finally, we verified the predicted eigenvalues by comparing the projected test error over the 1st PC, $\\boldsymbol{v_1}^T(\\boldsymbol{x} - \\mu_1(\\boldsymbol{y}))$, to the predicted 1st eigenvalue $\lambda_1$. The average ratio between those two quantities should theoretically be 1. For the 17 noisy images appearing in the manuscript, we found that the average of this ratio is 0.94. In the final version we will expand this experiment to a larger set of noisy test images.

Reviewer comment (Reproducibility):
While the authors claim the ability to compute the principal components (PCs) of the posterior distribution for any specified image region, the implementation details remain unclear to me. Seems there are no clear connection between the PCs and image features. At least it is not controllable.

Author response:
**“While the authors claim the ability to compute the principal components (PCs) of the posterior distribution for any specified image region, the implementation details remain unclear to me. Seems there are no clear connection between the PCs and image features. At least it is not controllable.”**

First, we would like to point out that our code is available in the provided anonymous repository, which might help mitigate any unclarity implementation-wise.

$\\boldsymbol{v}$ can be masked during Alg. 1, which makes sure that the computed PC affects only the desired region (set by the mask). This can be seen in the different widths and heights of the various PCs displayed in the experiments. This is an important point which we agree was not described clearly in the paper and will fix it in the final version. Thank you for pointing it out.

Reviewer comment (Theory):
The computational complexity should be reported in the revision.

Author response:
**”The computational complexity should be reported in the revision.”**

Our computational complexity is only dependent on the chosen number of subspace iterations $K$ and requires $K+1$ NFEs. In all experiments we used $K=50$, so that our method involved 51 NFEs per input image. Yet, as can be seen in Fig. S1, convergence is typically achieved after around 10 iterations, so that in principle the method could be run with only 11 NFEs.

Reviewer comment (Evaluation):
More importantly, absolute error to the ground-truth should be also presented in Fig. 4 and Fig. 5 to show that the uncertainty indeed can better reflect the restoration error.

Author response:
**“In this paper, no baseline methods are compared against with. No quantitative results about the uncertainty calibration such as expected calibration error etc. are reported.”, “Absolute error to the ground-truth should be also presented in Fig. 4 and Fig.5 to show that the uncertainty indeed can better reflect the restoration error.”**

Thanks for this important point. Please note that it is impossible to *directly* measure the quality of the estimated posterior PCs because denoising datasets contain only one clean image $\\boldsymbol{x}$ for each noisy image $\\boldsymbol{y}$. This single $\\boldsymbol{x}$ is just one sample from the posterior $p\_{\\mathbf{x}|\\mathbf{y}}$ and therefore it cannot be used to extract a “ground-truth” posterior covariance matrix or “ground-truth” PCs.

However, it is possible to compare our approach to a very strong baseline - that of diffusion-based posterior samplers, as mentioned above. We added this comparison to App. K. For each noisy image, we generated many posterior samples using DDNM[Wang et al., 2023] and used them to calculate the PCs of the posterior. As can be seen in Fig. S4, as the number of posterior samples increases, the PCs estimated using this baseline become cleaner and more similar to our PCs. However, note that even with 500 samples, the PCs of this baseline do not seem to have fully converged, and generating 500 posterior samples using DDNM requires 50,000 NFEs (just for analyzing the uncertainty for a single noisy input image!).

We also added a numerical comparison. Due to the heavy computations involved in generating posterior samples with DDNM, for now we conducted comparisons only with the 4 noisy images of Fig. 2, but we will include the results for a large dataset in the revised manuscript. The comparison is as follows. First, the empirical mean of the samples generated by DDNM should theoretically approximate the posterior mean, which is the MSE-optimal restoration. As we verify, this estimate is indeed very close to our denoiser’s output, and they both achieve practically the same MSE to the ground-truth images. Second, we computed the PCs from the posterior samples (after subtracting their mean). We compared those PCs to the PCs of our method by measuring the norm of the error after projecting it onto these PCs. The larger this norm, the larger the portion of the error that these PCs account for (larger is better). Mathematically, this measure is defined as $\\|\\boldsymbol{V}^T(\\boldsymbol{x} - \\mu\_1(\\boldsymbol{y}))\\|^2\_2$, where $\\boldsymbol{V}$ is a matrix containing the PCs as columns. We found that the mean of this measure for both methods is very close and ours is even slightly better: ours is 0.81 and the baseline’s is 0.63.

Finally, we verified the predicted eigenvalues by comparing the projected test error over the 1st PC, $\\boldsymbol{v_1}^T(\\boldsymbol{x} - \\mu_1(\\boldsymbol{y}))$, to the predicted 1st eigenvalue $\lambda_1$. The average ratio between those two quantities should theoretically be 1. For the 17 noisy images appearing in the manuscript, we found that the average of this ratio is 0.94. In the final version we will expand this experiment to a larger set of noisy test images.