Original: E VALUATING U NSUPERVISED D ENOISING R EQUIRES U NSUPERVISED M ETRICS
Revised: E VALUATING U NSUPERVISED D ENOISING R EQUIRES U NSUPERVISED M ETRICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised denoising is a crucial challenge in real-world imaging applications.
Revised: Unsupervised denoising is a crucial challenge in real-world imaging applications.

Original: Unsupervised deep-learning methods have demonstrated impressive performance on benchmarks based on synthetic noise.
Revised: Unsupervised deep-learning methods have demonstrated impressive performance on benchmarks based on synthetic noise.

Original: However, no metrics are available to evaluate these methods in an unsupervised fashion.
Revised: However, no metrics are available to evaluate these methods in an unsupervised fashion.

Original: This is highly problematic for the many practical applications where ground-truth clean images are not available.
Revised: This is highly problematic for the many practical applications where ground-truth clean images are not available.

Original: In this work, we propose two novel metrics: the unsupervised mean squared error (MSE) and the unsupervised peak signal-to-noise ratio (PSNR), which are computed using only noisy data.
Revised: In this work, we propose two novel metrics: the unsupervised mean squared error (MSE) and the unsupervised peak signal-to-noise ratio (PSNR), which are computed using only noisy data.

Original: We provide a theoretical analysis of these metrics, showing that they are asymptotically consistent estimators of the supervised MSE and PSNR.
Revised: We provide a theoretical analysis of these metrics, showing that they are asymptotically consistent estimators of the supervised MSE and PSNR.

Original: Controlled numerical experiments with synthetic noise confirm that they provide accurate approximations in practice.
Revised: Controlled numerical experiments with synthetic noise confirm that they provide accurate approximations in practice.

Original: We validate our approach on realworld data from two imaging modalities: videos in raw format and transmission electron microscopy.
Revised: We validate our approach on realworld data from two imaging modalities: videos in raw format and transmission electron microscopy.

Original: Our results demonstrate that the proposed metrics enable unsupervised evaluation of denoising methods based exclusively on noisy data.
Revised: Our results demonstrate that the proposed metrics enable unsupervised evaluation of denoising methods based exclusively on noisy data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image denoising is a fundamental challenge in image and signal processing, as well as a key preprocessing step for computer vision tasks.
Revised: Image denoising is a fundamental challenge in image and signal processing, as well as a key preprocessing step for computer vision tasks.

Original: Convolutional neural networks achieve state-of-the-art performance for this problem, when trained using databases of clean images corrupted with simulated noise Zhang et al.
Revised: Convolutional neural networks achieve state-of-the-art performance for this problem, when trained using databases of clean images corrupted with simulated noise Zhang et al.

Original: (2017a).
Revised: (2017a).

Original: However, in real-world imaging applications such as microscopy, noiseless ground truth videos are often not available.
Revised: However, in real-world imaging applications such as microscopy, noiseless ground truth videos are often not available.

Original: This has motivated the development of unsupervised denoising approaches that can be trained using only noisy measurements Lehtinen et al.
Revised: This has motivated the development of unsupervised denoising approaches that can be trained using only noisy measurements Lehtinen et al.

Original: Xie et al.
Revised: Xie et al.

Original: ; Laine et al.
Revised: ; Laine et al.

Original: Sheth et al.
Revised: Sheth et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: These methods have demonstrated impressive performance on natural-image benchmarks, essentially on par with the supervised state of the art.
Revised: These methods have demonstrated impressive performance on natural-image benchmarks, essentially on par with the supervised state of the art.

Original: However, to the best of our knowledge, no unsupervised metrics are currently available to evaluate them using only noisy data .
Revised: However, to the best of our knowledge, no unsupervised metrics are currently available to evaluate them using only noisy data .

Original: Reliance on supervised metrics makes it very challenging to create benchmark datasets using realworld measurements, because obtaining the ground-truth clean images required by these metrics is often either impossible or very constraining.
Revised: Reliance on supervised metrics makes it very challenging to create benchmark datasets using realworld measurements, because obtaining the ground-truth clean images required by these metrics is often either impossible or very constraining.

Original: In practice, clean images are typically estimated through temporal averaging, which suppresses dynamic information that is often crucial in scientific applications.
Revised: In practice, clean images are typically estimated through temporal averaging, which suppresses dynamic information that is often crucial in scientific applications.

Original: Consequently, quantitative evaluation of unsupervised denoising methods is currently almost completely dominated by natural image benchmark datasets with simulated noise Lehtinen et al.
Revised: Consequently, quantitative evaluation of unsupervised denoising methods is currently almost completely dominated by natural image benchmark datasets with simulated noise Lehtinen et al.

Original: ; Xie et al.
Revised: ; Xie et al.

Original: ; Laine et al.
Revised: ; Laine et al.

Original: Sheth et al.
Revised: Sheth et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021), which are not always representative of the signal and noise characteristics that arise in real-world imaging applications.
Revised: (2021), which are not always representative of the signal and noise characteristics that arise in real-world imaging applications.

Original: The lack of unsupervised metrics also limits the application of unsupervised denoising techniques in practice.
Revised: The lack of unsupervised metrics also limits the application of unsupervised denoising techniques in practice.

Original: In the absence of quantitative metrics, domain scientists must often rely on visual inspection to evaluate performance on real measurements.
Revised: In the absence of quantitative metrics, domain scientists must often rely on visual inspection to evaluate performance on real measurements.

Original: This is particularly restrictive for deep-learning approaches, because it makes it impossible to perform systematic hyperparameter optimization and model selection on the data of interest.
Revised: This is particularly restrictive for deep-learning approaches, because it makes it impossible to perform systematic hyperparameter optimization and model selection on the data of interest.

Original: In this work, we propose two novel unsupervised metrics to address these issues: the unsupervised mean-squared error (uMSE) and the unsupervised peak signal-to-noise ratio (uPSNR), which are computed exclusively from noisy data .
Revised: In this work, we propose two novel unsupervised metrics to address these issues: the unsupervised mean-squared error (uMSE) and the unsupervised peak signal-to-noise ratio (uPSNR), which are computed exclusively from noisy data .

Original: These metrics build upon existing unsupervised denoising methods, which minimize an unsupervised cost function equal to the difference between the denoised estimate and additional noisy copies of the signal of interest Lehtinen et al.
Revised: These metrics build upon existing unsupervised denoising methods, which minimize an unsupervised cost function equal to the difference between the denoised estimate and additional noisy copies of the signal of interest Lehtinen et al.

Original: The uMSE is equal to this cost function modified with a correction term, which renders it an unbiased estimator of the supervised MSE.
Revised: The uMSE is equal to this cost function modified with a correction term, which renders it an unbiased estimator of the supervised MSE.

Original: We provide a theoretical analysis of the uMSE and uPSNR, proving that they are asymptotically consistent estimators of the supervised MSE and PSNR respectively.
Revised: We provide a theoretical analysis of the uMSE and uPSNR, proving that they are asymptotically consistent estimators of the supervised MSE and PSNR respectively.

Original: Controllednumerical experiments on supervised benchmarks, where the true MSE and PSNR can be computed exactly, confirm that the uMSE and uPSNR provide accurate approximations in practice.
Revised: Controlled experiments on supervised benchmarks, where the true MSE and PSNR can be computed exactly, confirm that the uMSE and uPSNR provide accurate approximations.

Original: In addition, we validate the metrics on a dataset of videos in RAW format, contaminated with real noise Yue et al.
Revised: In addition, we validate the metrics on video data in RAW format, contaminated with real noise that does not follow a known predefined model.

Original: In order to illustrate the potential impact of the proposed metrics on imaging applications where no ground-truth is available, we apply them to transmission-electron-microscopy (TEM) data.
Revised: In order to illustrate the potential impact of the proposed metrics on imaging applications where no ground-truth is available, we apply them to transmission-electron-microscopy (TEM) data.

Original: Recent advances in direct electron detection systems make it possible for experimentalists to acquire highly time-resolved movies of dynamic events at frame rates in the kilohertz range Faruqi & McMullan (2018); Ercius et al.
Revised: Recent advances in direct electron detection systems make it possible for experimentalists to acquire highly time-resolved movies of dynamic events at frame rates in the kilohertz range Faruqi & McMullan (2018); Ercius et al.

Original: (2020), which is critical to advance our understanding of functional materials.
Revised: (2020), which is critical to advance our understanding of functional materials.

Original: Acquisition at such high temporal resolution results in severe degradation by shot noise.
Revised: Acquisition at such high temporal resolution results in severe degradation by shot noise.

Original: We show that unsupervised methods based on deep learning can be effective in removing this noise, and that our proposed metrics can be used to evaluate their performance quantitatively using only noisy data.
Revised: We show that unsupervised methods based on deep learning can be effective in removing this noise, and that our proposed metrics can be used to evaluate their performance quantitatively using only noisy data.

Original: To summarize, our contributions are (1) two novel unsupervised metrics presented in Section 3, (2) a theoretical analysis providing an asymptotic characterization of their statistical properties (Section 4), (3) experiments showing the accuracy of the metrics in a controlled situation where ground-truth clean images are available (Section 5), (4) validation on real-world videos in RAW format (Section 6), and (5) an application to a real-world electron-microscopy dataset, which illustrates the challenges of unsupervised denoising in scientific imaging (Section 7).
Revised: To summarize, our contributions are (1) two novel unsupervised metrics presented in Section 3, (2) a theoretical analysis providing an asymptotic characterization of their statistical properties (Section 4), (3) experiments showing the accuracy of the metrics in a controlled situation where ground-truth clean images are available (Section 5), (4) validation on real-world videos in RAW format (Section 6), and (5) an application to a real-world electron-microscopy dataset, which illustrates the challenges of unsupervised denoising in scientific imaging (Section 7).

Original:  B ACKGROUND AND R ELATED WORK
Revised:  B ACKGROUND AND R ELATED WORK

Original: Unsupervised denoising The past few years have seen ground-breaking progress in unsupervised denoising, pioneered by Noise2Noise, a technique where a neural network is trained on pairs of noisy imagescorresponding to the same clean image Lehtinen et al.
Revised: Unsupervised denoising The past few years have seen ground-breaking progress in unsupervised denoising, pioneered by Noise2Noise, a technique where a neural network is trained on pairs of noisy images Lehtinen et al.

Original: Our unsupervised metrics are inspired by Noise2Noise, which optimizes a cost function equal to our proposed unsupervised MSE, but without a correction term (which is not needed for training models).
Revised: Our unsupervised metrics are inspired by Noise2Noise, which optimizes a cost function equal to our proposed unsupervised MSE, but without a correction term (which is not needed for training models).

Original: Subsequent work focused on performing unsupervised denoising from single images using variations of the blind-spot method, where a model is trained to estimate each noisy pixel value using its neighborhood but not the noisy pixel itself (to avoid the trivial identity solution)
Revised: Subsequent work focused on performing unsupervised denoising from single images using variations of the blind-spot method, where a model is trained to estimate each noisy pixel value using its neighborhood but not the noisy pixel itself (to avoid the trivial identity solution)

Original: Krull et al.
Revised: Krull et al.

Original: ; Laine et al.
Revised: ; Laine et al.

Original: ; Batson & Royer (2019); Sheth et al.
Revised: ; Batson & Royer (2019); Sheth et al.

Original: ; Xie et al.
Revised: ; Xie et al.

Original: (2020), or Stein’s unbiased risk estimator of the MSE (in the case of additive Gaussian noise) Raphan & Simoncelli (2008);

Original: Metzler et al.

Original: (2018); Soltanayev & Chun (2018); Mohan et al.

Original: More recently, Neighbor2Neighbor revisited the Noise2Noise method, generating noisy image pairs from a single noisy image via spatial subsampling Huang et al.
Revised: More recently, Neighbor2Neighbor revisited the Noise2Noise method, generating noisy image pairs from a single noisy image via spatial subsampling Huang et al.

Original: (2021), an insight that can also be leveraged in combination with our proposed metrics, as explained inSection B.
Revised: (2021), an insight that can also be leveraged in combination with our proposed metrics, as explained in

Original: Existing evaluation approaches In the literature, quantitative evaluation of unsupervised denoising techniques has mostly relied on images and videos corrupted with synthetic noise Lehtinen et al.
Revised: Section C. Our contribution with respect to these methods is a novel unsupervised metric that can be used for evaluation , as it is designed to be an unbiased and consistent estimator of the MSE. Stein’s unbiased risk estimator (SURE) provides an asymptotically unbiased estimator of the MSE for i.i.d. Gaussian noise Donoho & Johnstone (1995). This cost function has been used for training unsupervised denoisers Metzler et al.

Revised: (2018); Soltanayev & Chun (2018); Zhussip et al.

Revised: Mohan et al.

Revised: In principle, SURE could be used to compute the MSE for evaluation, but it has certain limitations: (1) a closed form expression of the noise likelihood is required, including the value of the noise parameters (for example, this is not known for the real-world datasets in Sectionsand 7), (2) computing SURE requires approximating the divergence of a denoiser (usually via Monte

Revised: Carlo methods Ramani et al.

Revised: (2008)), which is computationally very expensive.

Revised: Developing practical unsupervised metrics based on SURE and studying their theoretical properties is an interesting direction for future research.

Revised: Existing evaluation approaches In the literature, quantitative evaluation of unsupervised denoising techniques has mostly relied on images and videos corrupted with synthetic noise Lehtinen et al.

Original: ; Krull et al.
Revised: ; Krull et al.

Original: ; Laine et al.
Revised: ; Laine et al.

Original: ; Batson & Royer (2019); Sheth et al.
Revised: ; Batson & Royer (2019); Sheth et al.

Original: ; Xie et al.
Revised: ; Xie et al.

Original: Recently, a few datasets containing real noisy data have been created Abdelhamed et al.
Revised: Recently, a few datasets containing real noisy data have been created Abdelhamed et al.

Original: (2018); Plotz & Roth (2017);
Revised: (2018); Plotz & Roth (2017);

Original: Xu et al.
Revised: Xu et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: Evaluation on these datasets is based on supervised MSE and PSNR computed from estimated clean images obtained by averaging multiple noisy frames.
Revised: Evaluation on these datasets is based on supervised MSE and PSNR computed from estimated clean images obtained by averaging multiple noisy frames.

Original: Unfortunately, as a result, the metrics cannot capture dynamically-changing features, which are of interest in many applied domains.
Revised: Unfortunately, as a result, the metrics cannot capture dynamically-changing features, which are of interest in many applied domains.

Original: In addition, unless the signal-to-noise ratio is quite high, it is necessary to average over a large number of frames toaccurately approximate thesupervised MSE.
Revised: In addition, unless the signal-to-noise ratio is quite high, it is necessary to average over a large number of frames to approximate the MSE.

Original: For example, as explained in Section C, for CNN based denoiser and an image corrupted by additive Gaussian noise with standard deviation σ= 15 we need to average >noisy images toachieve the same approximation accuracy as our proposed approach (see Figure 9), which only requires 3 noisy images, and can also be computed from a single noisy image (via spatial subsampling, as explained in Section B).
Revised: For example, as explained in Section E, for CNN based denoiser and an image corrupted by additive Gaussian noise with standard deviation σ 15 we need to average > 1500 noisy images to our proposed approach (see Figure 10), only Noise-Level Estimation .

Revised: The correction term in uMSE can be interpreted as an estimate of the noise level, obtained by cancelling out the clean signal.

Revised: In this sense, it is related to noise-level estimation methods Liu et al.

Original: U NSUPERVISED M ETRICS F OR U NSUPERVISED D ENOISING 3. T HE U NSUPERVISED M EAN S QUARED E RROR
Revised: ; Lebrun et al. ; Arias & Morel (2018). However, unlike uMSE, these methods typically assume a parametric model for the noise, and are not used for evaluation. No-reference image quality assessment methods evaluate the perceptual quality of an image Li (2002); Mittal et al.

Revised: (2012), but not whether it is consistent with an underlying ground-truth corresponding to the observed noisy measurements, which is the goal of our proposed metrics.

Revised:  U NSUPERVISED M ETRICS F OR U NSUPERVISED D ENOISING 3. T HE U NSUPERVISED M EAN S QUARED E RROR

Original: The goal of denoising is to estimate a clean signal from noisy measurements.
Revised: The goal of denoising is to estimate a clean signal from noisy measurements.

Original: Let x ∈ R n be a signal or a set of signals with n total entries.
Revised: Let x ∈ R n be a signal or a set of signals with n total entries.

Original: We denote the corresponding noisy data by y ∈ R n .
Revised: We denote the corresponding noisy data by y ∈ R n .

Original: A denoiser f : R n → R n is a function that maps the input y to an estimate of x .
Revised: A denoiser f : R n → R n is a function that maps the input y to an estimate of x .

Original: A common metric to evaluate the quality of a denoiser is the mean squared error between the clean signal and the estimate,
Revised: A common metric to evaluate the quality of a denoiser is the mean squared error between the clean signal and the estimate,

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, in most real-world scenarios clean ground-truth signals are not available and evaluation can only be carried out in an unsupervised fashion, i.e. exclusively from the noisy measurements .
Revised: Unfortunately, in most real-world scenarios clean ground-truth signals are not available and evaluation can only be carried out in an unsupervised fashion, i.e. exclusively from the noisy measurements .

Original: In this section we propose an unsupervised estimator of MSE inspired by recent advances in unsupervised denoising Lehtinen et al.
Revised: In this section we propose an unsupervised estimator of MSE inspired by recent advances in unsupervised denoising Lehtinen et al.

Original: The key idea is to compare the denoised signal to a noisy reference , which corresponds to the same clean signal corrupted by independent noise.
Revised: The key idea is to compare the denoised signal to a noisy reference , which corresponds to the same clean signal corrupted by independent noise.

Original: In order to motivate our approach, let us assume that the noise is additive, so that y := x + z for a zero-mean noise vector z ∈ R n .
Revised: In order to motivate our approach, let us assume that the noise is additive, so that y := x + z for a zero-mean noise vector z ∈ R n .

Original: Imagine that we have access to a noisy reference a := x + w corresponding to the same underlying signal x , but corrupted with a different noise realization w ∈ R n independent from z (Section 3.3 explains how to obtain such references in practice).
Revised: Imagine that we have access to a noisy reference a := x + w corresponding to the same underlying signal x , but corrupted with a different noise realization w ∈ R n independent from z (Section 3.3 explains how to obtain such references in practice).

Original: The mean squared difference between the denoised estimate and the reference is approximately equal to the sum of the MSE and the variance σ 2 of the noise,
Revised: The mean squared difference between the denoised estimate and the reference is approximately equal to the sum of the MSE and the variance σ 2 of the noise,

Original: [Equation]
Revised: [Equation]

Original: because the cross-term 1 n (cid:80) ni =1 w i ( x i − f
Revised: because the cross-term 1 n (cid:80) ni =1 w i ( x i − f

Original: ( y ) i ) 2 cancels out if w i and y i (and hence f
Revised: ( y ) i ) 2 cancels out if w i and y i (and hence f

Original: ( y i ) ) are independent (and the mean of the noise is zero).
Revised: ( y i ) ) are independent (and the mean of the noise is zero).

Original: Approximations to equation 3 are used by different unsupervised methods to train neural networks for denoising Lehtinen et al.
Revised: Approximations to equation 3 are used by different unsupervised methods to train neural networks for denoising Lehtinen et al.

Original: ; Xie et al.
Revised: ; Xie et al.

Original: ; Laine et al.
Revised: ; Laine et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: The noise term 1 n (cid:80) ni
Revised: The noise term 1 n (cid:80) ni

Original: =

Revised: =

Original: w 2 i in equation 3 is not problematic for training denoisers as long as it is independent from the input y .
Revised: w 2 i in equation 3 is not problematic for training denoisers as long as it is independent from the input y .

Original: However, it is definitely problematic for evaluating denoisers.
Revised: However, it is definitely problematic for evaluating denoisers.

Original: In order to estimate the MSE we need to cancel it out.
Revised: In order to estimate the MSE we need to cancel it out.

Original: We propose to achieve this by using two other noisy references b := x + v and c := x + u , which are noisy measurements corresponding to the clean signal x , but corrupted with different, independent noise realizations v and u (just like a ).
Revised: We propose to achieve this by using two other noisy references b := x + v and c := x + u , which are noisy measurements corresponding to the clean signal x , but corrupted with different, independent noise realizations v and u (just like a ).

Original: Subtracting these references and dividing by two yields an estimate of the noise variance,
Revised: Subtracting these references and dividing by two yields an estimate of the noise variance,

Original: [Equation]
Revised: [Equation]

Original: which can then be subtracted from equation 3 to estimate the MSE.
Revised: which can then be subtracted from equation 3 to estimate the MSE.

Original: This yields our proposed unsupervised metric, which we call unsupervised mean squared error (uMSE), depicted in Figure 1.
Revised: This yields our proposed unsupervised metric, which we call unsupervised mean squared error (uMSE), depicted in Figure 1.

Original: [Figure]

Original: Definition 1 (Unsupervised mean squared error) .
Revised: Definition 1 (Unsupervised mean squared error) .

Original: Given a noisy input signal y ∈ R n and three noisy references a , b , c ∈ R n the unsupervised mean squared error of a denoiser f : R n → R n is
Revised: Given a noisy input signal y ∈ R n and three noisy references a , b , c ∈ R n the unsupervised mean squared error of a denoiser f : R n → R n is

Original: [Equation]
Revised: [Equation]

Original: Theorem 4 in Section 4 establishes that the uMSE is a consistent estimator of the MSE as long as (1) the noisy input and the noisy references are independent, (2) their means equal the corresponding entries of the ground-truth clean signal, and (3) their higher-order moments are bounded.
Revised: Theorem 4 in Section 4 establishes that the uMSE is a consistent estimator of the MSE as long as (1) the noisy input and the noisy references are independent, (2) their means equal the corresponding entries of the ground-truth clean signal, and (3) their higher-order moments are bounded.

Original: These conditions are satisfied by most noise models of interest in signal and image processing, such as
Revised: These conditions are satisfied by most noise models of interest in signal and image processing, such as

Original: Poisson shot noise or additive Gaussian noise.
Revised: Poisson shot noise or additive Gaussian noise.

Original: In Section 3.3 we address the question of how to obtain the noisy references required to estimate the uMSE.
Revised: In Section 3.3 we address the question of how to obtain the noisy references required to estimate the uMSE.

Original: Section A explains how to compute confidence intervals for the uMSE via bootstrapping.
Revised: Section B explains how to compute confidence intervals for the uMSE via bootstrapping.

Original: 3. T HE U NSUPERVISED P EAK S IGNAL -T O -N OISE R ATIO
Revised: 3. T HE U NSUPERVISED P EAK S IGNAL -T O -N OISE R ATIO

Original: Peak signal-to-noise ratio (PSNR) is currently the most popular metric to evaluate denoising quality.
Revised: Peak signal-to-noise ratio (PSNR) is currently the most popular metric to evaluate denoising quality.

Original: It is a logarithmic function of MSE defined on a decibel scale,
Revised: It is a logarithmic function of MSE defined on a decibel scale,

Original: [Equation]
Revised: [Equation]

Original: where M is a fixed constant representing the maximum possible value of the signal of interest, which is usually set equal to 255 for images.
Revised: where M is a fixed constant representing the maximum possible value of the signal of interest, which is usually set equal to 255 for images.

Original: Our definition of uMSE can be naturally extended to yield an unsupervised PSNR (uPSNR).
Revised: Our definition of uMSE can be naturally extended to yield an unsupervised PSNR (uPSNR).

Original: Definition 2 (Unsupervised peak signal-to-noise ratio) .
Revised: Definition 2 (Unsupervised peak signal-to-noise ratio) .

Original: Given a noisy input signal y ∈ R n and three noisy references a , b , c ∈ R n the peak signal-to-noise ratio of a denoiser f : R n → R n is
Revised: Given a noisy input signal y ∈ R n and three noisy references a , b , c ∈ R n the peak signal-to-noise ratio of a denoiser f : R n → R n is

Original: [Equation]
Revised: [Equation]

Original: where M is the maximum possible value of the signal of interest.
Revised: where M is the maximum possible value of the signal of interest.

Original: Corollary 5 establishes that the uPSNR is a consistent estimator of the PSNR, under the same conditions that guarantee consistency of the uMSE.
Revised: Corollary 5 establishes that the uPSNR is a consistent estimator of the PSNR, under the same conditions that guarantee consistency of the uMSE.

Original: Section A explains how to compute confidence intervals for the uPSNR via bootstrapping.
Revised: Section B explains how to compute confidence intervals for the uPSNR via bootstrapping.

Original: [Figure]
Revised: [Figure]

Original: 3. C OMPUTING N OISY R EFERENCES I N P RACTICE
Revised: 3. C OMPUTING N OISY R EFERENCES I N P RACTICE

Original: Our proposed metrics rely on the availability of three noisy references, which ideally should correspond to the same clean image contaminated with independent noise.
Revised: Our proposed metrics rely on the availability of three noisy references, which ideally should correspond to the same clean image contaminated with independent noise.

Original: Deviations between the clean signal in each reference violate Condition 2 in Section  andtherefore introduce a bias in theproposed metrics.
Revised: Deviations between the clean signal in each reference violate Condition 2 in Section 4, and introduce a bias in the metrics.

Original: The references should therefore be constructed to minimize this deviation.

Original: We propose two approaches to compute the references in practice, illustrated in Figure 2.
Revised: We propose two approaches to compute the references in practice, illustrated in Figure 5.

Original: Using consecutive frames acquired within a short time interval is appropriate if there is no dynamic content that changes from frame to frame.
Revised: Multiple images: The references can be computed from consecutive frames acquired within a short time interval. This approach is preferable for datasets where the image content does not experience rapid dynamic changes from frame to frame.

Original: We apply this approach to the RAW videos in Section 6, where the content is static.
Revised: We apply this approach to the RAW videos in Section 6, where the content is static.

Original: Applying spatial subsampling to a single image , as described in Section 
Revised: Single image: The references can be computed from a single image via spatial subsampling, as described in Section C. Section C shows that this approach is effective as long as the image content is sufficiently smooth with respect to the pixel resolution.

Original: B. We apply this approach to the electron-microscopy data in Section 7, where preserving dynamic content is important.
Revised: We apply this approach to the electronmicroscopy data in Section 7, where preserving dynamic content is important.

Original: We study the bias introduced

Original: by spatial subsampling in Figure 4 (see also Figure 8).

Original:  S TATISTICAL P ROPERTIES OF THE P ROPOSED M ETRICS
Revised:  S TATISTICAL P ROPERTIES OF THE P ROPOSED M ETRICS

Original: In this section, we establish that the proposed unsupervised metrics provide a consistent estimate of the MSE and PSNR.
Revised: In this section, we establish that the proposed unsupervised metrics provide a consistent estimate of the MSE and PSNR.

Original: In our analysis, the ground truth signal or set of signals is represented as a deterministic vector x ∈ R n .
Revised: In our analysis, the ground truth signal or set of signals is represented as a deterministic vector x ∈ R n .

Original: The corresponding noisy data are also modeled as a deterministic vector y ∈ R n that is fed into a denoiser f : R n → R n to produce the denoised estimate f ( y ) .
Revised: The corresponding noisy data are also modeled as a deterministic vector y ∈ R n that is fed into a denoiser f : R n → R n to produce the denoised estimate f ( y ) .

Original: The
Revised: The

Original: MSE of the estimate is a deterministic quantity equal to
Revised: MSE of the estimate is a deterministic quantity equal to

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original:  The uMSE estimator in Definition 1 depends on three noisy references ˜ a , ˜ b , ˜ c , which we model as random variables.
Revised: Noise Model. The uMSE estimator in Definition 1 depends on three noisy references ˜ a , ˜ b , ˜ c , which we model as random variables.

Original: 1 Our analysis assumes that these random variables satisfy two conditions:
Revised: 1 Our analysis assumes that these random variables satisfy two conditions:

Original: Condition 1 (independence): The entries of ˜ a , ˜ b , ˜ c are all mutually independent.
Revised: Condition 1 (independence): The entries of ˜ a , ˜ b , ˜ c are all mutually independent.

Original: Condition 2 (centered noise): The mean of the i th entry of ˜ a , ˜ b , ˜ c equals the corresponding entry of the clean signal, E [˜ a i ] =
Revised: Condition 2 (centered noise): The mean of the i th entry of ˜ a , ˜ b , ˜ c equals the corresponding entry of the clean signal, E [˜ a i ] =

Original: E [˜ b i ] =
Revised: E [˜ b i ] =

Original: E [˜ c i ] = x i , 1 ≤ i ≤ n .
Revised: E [˜ c i ] = x i , 1 ≤ i ≤ n .

Original: Two popular noise models that satisfy these conditions are:
Revised: Two popular noise models that satisfy these conditions are:

Original: • Additive Gaussian , where ˜ a
Revised: • Additive Gaussian , where ˜ a

Original: i := x i + ˜ w i , ˜ b
Revised: i := x i + ˜ w i , ˜ b

Original: i := x i + ˜ v i , ˜ c
Revised: i := x i + ˜ v i , ˜ c

Original: i := x i + ˜ u i , for i.i
Revised: i := x i + ˜ u i , for i.i.d.

Original: d. Gaussian ˜ w i , ˜ v i , ˜ u i .
Revised: Gaussian ˜ w i , ˜ v i , ˜ u i .

Original: • Poisson , where ˜ a
Revised: • Poisson , where ˜ a

Original: i , ˜ b i , ˜ c
Revised: i , ˜ b i , ˜ c

Original: i are i.i.d.
Revised: i are i.i.d.

Original: Poisson random variables with parameter x i .
Revised: Poisson random variables with parameter x i .

Original:  Our goal is to study the statistical properties of the uMSEcomputed from the noisy references, (cid:94) uMSE := 1 n n (cid:88)
Revised: Theoretical Guarantees. Our goal is to study the statistical properties of the uMSE (cid:94) uMSE [Equation] =

Original: i = (cid:103) uSE i , (cid:103) uSE i :=
Revised: [Equation] n n (cid:88) [Equation] = (cid:103) uSE i , (cid:103) uSE i :=

Original: (˜
Revised: (˜

Original: [Equation]
Revised: [Equation]

Original: As indicated by the tilde, under our modeling assumptions, the uMSE is a random variable.
Revised: As indicated by the tilde, under our modeling assumptions, the uMSE is a random variable.

Original: We first show that the correction factor in the definition of uMSE succeeds in debiasing the estimator, so that its mean is equal to the MSE.
Revised: We first show that the correction factor in the definition of uMSE succeeds in debiasing the estimator, so that its mean is equal to the MSE.

Original: Theorem 3 (The uMSE is unbiased, proof in Section E.1) .
Revised: Theorem 3 (The uMSE is unbiased, proof in Section G.1) .

Original: If Conditions 1 and 2 hold, the uMSE is an unbiased estimator of the MSE, i.e. E (cid:104) (cid:94) uMSE (cid:105) =
Revised: If Conditions 1 and 2 hold, the uMSE is an unbiased estimator of the MSE, i.e. E [ (cid:94) uMSE] =

Original: MSE .
Revised: MSE .

Original: Theorem 3 establishes that the distribution of the uMSE is centered at the MSE.
Revised: Theorem 3 establishes that the distribution of the uMSE is centered at the MSE.

Original: We now show that its variancearound the MSE shrinks at a rate inversely proportional to the number of signal entries n , and therefore converges to the MSE in mean square and probability as n → ∞ (see Figurefor a numerical demonstration).
Revised: We now show that its variance shrinks at a rate inversely proportional to the number of signal entries n , and therefore converges to the MSE in mean square and probability as n → ∞ (see Figure 2 for a numerical demonstration).

Original: This occurs as long as the higher central moments of noise and the entrywise denoising error are bounded by a constant, which is to be expected in most realistic scenarios.
Revised: This occurs as long as the higher central moments of noise and the entrywise denoising error are bounded by a constant, which is to be expected in most realistic scenarios.

Original: Theorem 4 (The uMSE is consistent, proof in Section E.2) .
Revised: Theorem 4 (The uMSE is consistent, proof in Section G.2) .

Original: Let µ [ k ] i denote the k th central moment of ˜ a i , ˜ b
Revised: Let µ [ k ] i denote the k th central moment of ˜ a i , ˜ b

Original: i , ˜ c
Revised: i , ˜ c

Original: i , and γ := max 1 ≤ i ≤ n | x i − f ( y ) i | the maximum entrywise denoising error.
Revised: i , and γ := max 1 ≤ i ≤ n | x i − f ( y ) i | the maximum entrywise denoising error.

Original: If Conditions 1 and 2 hold, and there exists a constant α such that max 1 ≤ i ≤ n max (cid:110) µ [4] i , µ [3] i γ, γ 4 (cid:111) ≤ α , then the
Revised: If Conditions 1 and 2 hold, and there exists a constant α such that max 1 ≤ i ≤ n max (cid:110) µ [4] i , µ [3] i γ, γ 4 (cid:111) ≤ α , then the

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: Consequently, lim n [Equation] [Equation] ] = 0 , so the uMSE converges to the MSE in mean square and therefore also in probability.
Revised: Consequently, lim n →∞ E [( (cid:94) uMSE − MSE) 2 ] = 0 , so the uMSE converges to the MSE in mean square and therefore also in probability.

Original: Consistency of the uMSE implies consistency of the uPSNR.
Revised: Consistency of the uMSE implies consistency of the uPSNR.

Original: Corollary 5 (The uPSNR is consistent, proof in Section E.3) .
Revised: Corollary 5 (The uPSNR is consistent, proof in Section G.3) .

Original: Under the assumptions of Theorem 4, the uPSNR defined as
Revised: Under the assumptions of Theorem 4, the uPSNR defined as

Original: [Equation]
Revised: [Equation]

Original: where M is a fixed constant, converges in probability to the PSNR, as n → ∞ .
Revised: where M is a fixed constant, converges in probability to the PSNR, as n → ∞ .

Original: Thefollowing theorem establishes that the uMSE converges to a Gaussian random variable asymptotically as n → ∞ .
Revised: The uMSE converges to a Gaussian random variable asymptotically as n → ∞ .

Original: Theorem 6 (The uMSE is asymptotically normally distributed, proof in Section E.4) .
Revised: Theorem 6 (The uMSE is asymptotically normally distributed, proof in Section G.4) .

Original: If the first six central moments of ˜ a i , ˜ b
Revised: If the first six central moments of ˜ a i , ˜ b

Original: i , ˜ c
Revised: i , ˜ c

Original: i and the maximum entrywise denoising error max 1 ≤ i ≤ n | x i − f ( y ) i | are bounded, and Conditions 1 and 2 hold, the uMSE is asymptotically normally distributed as n → ∞ .
Revised: i and the maximum entrywise denoising error max 1 ≤ i ≤ n | x i − f ( y ) i | are bounded, and Conditions 1 and 2 hold, the uMSE is asymptotically normally distributed as n → ∞ .

Original: Our numerical experiments show that the distribution of the uMSE is well approximated as Gaussian even for relatively small values of n (see Figure 3).
Revised: Our numerical experiments show that the distribution of the uMSE is well approximated as Gaussian even for relatively small values of n (see Figure 2).

Original: This can be exploited to build confidence intervals for the uMSE and uPSNR, as explained in Section A.
Revised: This can be exploited to build confidence intervals for the uMSE and uPSNR, as explained in Section B.

Original:  C ONTROLLED E VALUATION OF THE P ROPOSED M ETRICS
Revised:  C ONTROLLED E VALUATION OF THE P ROPOSED M ETRICS

Original: In this section, we study the properties of the uMSE and uPSNR through numerical experiments in a controlled scenario where the ground-truth clean images are known.
Revised: In this section, we study the properties of the uMSE and uPSNR through numerical experiments in a controlled scenario where the ground-truth clean images are known.

Original: We use a dataset of natural images (Martin et al., 2001; Zhang et al., 2017b; Franzen, 1993) corrupted with additive Gaussian noise, and a dataset of simulated electron-microscopy images Vincent et al.
Revised: We use a dataset of natural images (Martin et al., 2001; Zhang et al., 2017b; Franzen, 1993) corrupted with additive Gaussian noise, and a dataset of simulated electron-microscopy images Vincent et al.

Original: (2021) corrupted with
Revised: (2021) corrupted with

Original: Poisson noise.
Revised: Poisson noise.

Original: For the two datasets, we compute the supervised MSE and PSNR using the groundtruth clean image.
Revised: For the two datasets, we compute the supervised MSE and PSNR using the groundtruth clean image.

Original: To compute the uMSE and uPSNR we use noisy references corresponding to the same clean image corrupted with independent noise.
Revised: To compute the uMSE and uPSNR we use noisy references corresponding to the same clean image corrupted with independent noise.

Original: We also compute the uMSE and uPSNR[Table] using noisy references obtained from a single noisy image via spatial subsampling, as described inSection B
Revised: We also compute the uMSE and uPSNR using noisy references obtained from a single noisy image via spatial subsampling, as described in

Original: [Table]
Revised: [Table]

Original:  which we denote by uMSE S and uPSNR S respectively.
Revised: Section C, which we denote by uMSE S and uPSNR S respectively.

Original: 2 All metrics are applied to multiple denoising approches, as described in more detail in Section F
Revised: 2 All metrics are applied to multiple denoising approches, as described in more detail in Section H

Original: [Table]
Revised: [Table]

Original: The results are reported in Tables 1 and 3.
Revised: The results are reported in Tables 1 and 3.

Original: When the noisy references correspond exactly to the same clean image (and therefore satisfy the conditions in Section 4), the unsupervised metrics are extremely accurate across different noise levels for all denoising methods.
Revised: When the noisy references correspond exactly to the same clean image (and therefore satisfy the conditions in Section 4), the unsupervised metrics are extremely accurate across different noise levels for all denoising methods.

Original:  When the noisy references are computed via spatial subsampling, the metrics are still very accurate for the electron-microscopy dataset, but less so for the natural-image dataset if the PSNR is high (above 20 dB).
Revised: Single-image results: When the noisy references are computed via spatial subsampling, the metrics are still very accurate for the electron-microscopy dataset, but less so for the natural-image dataset if the PSNR is high (above 20 dB).

Original: The culprit is the difference between the clean images corresponding to each noisy referenceproduced via spatial subsampling (see Figure 2), which introduces a bias in the unsupervised metric, depicted in Figure 4.
Revised: The culprit is the difference between the clean images underlying each noisy reference (see Figure 5), which introduces a bias in the unsupervised metric, depicted in Figure 3.

Original: Figure 8 shows that the difference is more pronounced in natural images than in electron-microscopy images, which are smoother and have higher spatial resolution.
Revised: Figure 8 shows that the difference is more pronounced in natural images than in electron-microscopy images, which are smoother with respect to the pixel resolution (see Section C).

Original:  A PPLICATION TO V IDEOS IN RAW F ORMAT
Revised:  A PPLICATION TO V IDEOS IN RAW F ORMAT

Original: We evaluate our proposed metrics on a dataset of videos in raw format, consisting of direct readings from the sensor of a surveillance camera contaminated with real noise at five different ISO levels Yue et al.
Revised: We evaluate our proposed metrics on a dataset of videos in raw format, consisting of direct readings from the sensor of a surveillance camera contaminated with real noise at five different ISO levels Yue et al.

Original: The dataset contains 11 unique videos divided into 7 segments, each consisting ofnoisy frames that capture the same static object.
Revised: The dataset contains 11 unique videos divided into 7 segments, each consisting ofnoisy frames that capture the same static object.

Original: We consider four different denoisers: a wavelet-based method, temporal averaging and two versions of a state-of-the-art unsupervised deep-learning method using images and videos respectively Sheth et al.
Revised: We consider four different denoisers: a wavelet-based method, temporal averaging and two versions of a state-of-the-art unsupervised deep-learning method using images and videos respectively Sheth et al.

Original: A detailed description of the experiments is provided in Section G. Tables 4 and 2 compare our proposed unsupervised metrics (computed using three noisy frames in each segment) with MSE and PSNR estimates obtained via averaging from ten noisy frames.
Revised: A detailed description of the experiments is provided in Section I. Tables 4 and 2 compare our proposed unsupervised metrics (computed using three noisy frames in each segment) with MSE and PSNR estimates obtained via averaging from ten noisy frames.

Original: The two types of metric yield similar results: the deep-learning methods clearly outperform the other baselines, and the video-based methods outperform the image-based methods.
Revised: The two types of metric yield similar results: the deep-learning methods clearly outperform the other baselines, and the video-based methods outperform the image-based methods.

Original: As explained in Section C, the averaging-based MSE and PSNR are not consistent estimators of the true MSE and PSNR, and can be substantially less accurate than the uMSE and uPSNR (see Figure 9), so they should not be considered ground-truth metrics.
Revised: As explained in Section E, the averaging-based MSE and PSNR are not consistent estimators of the true MSE and PSNR, and can be substantially less accurate than the uMSE and uPSNR (see Figure 10), so they should not be considered ground-truth metrics.

Original:  A PPLICATION T O E LECTRON M ICROSCOPY
Revised:  A PPLICATION T O E LECTRON M ICROSCOPY

Original: Our proposedunsupervised metrics provide quantitative evaluation of denoising methods on data where no ground-truth clean images are available.
Revised: Our proposed metrics enable quantitative evaluation of denoisers in the absence of ground-truth clean images.

Original: We showcase this for transmission electron microscopy (TEM), a key imaging modality in material sciences.
Revised: We showcase this for transmission electron microscopy (TEM), a key imaging modality in material sciences.

Original: Recent developments enable the acquisition of high frame-rate images, capturing high temporal resolution dynamics, thought to be crucial in catalytic processes Crozier et al.
Revised: Recent developments enable the acquisition of high frame-rate images, capturing high temporal resolution dynamics, thought to be crucial in catalytic processes Crozier et al.

Original: Images acquired under these conditions are severely limited by noise.
Revised: Images acquired under these conditions are severely limited by noise.

Original: Recent work suggest that deep learning methods provide an effective[Figure] solution Sheth et al.
Revised: Recent work suggest that deep learning methods provide an effective solution Sheth et al.

Original: ; Mohan et al.
Revised: ; Mohan et al.

Original: (2022; 2021), but, instead of quantitative metrics, evaluation on real data has been limited to visual inspection. 
Revised: (2022; 2021), but, instead of quantitative metrics, evaluation on real data has been limited to visual inspection. [Figure]

Original: The TEM dataset consists of 18,597 noisy frames depicting platinum nanoparticles on a cerium oxide support.
Revised: The TEM dataset consists of 18,597 noisy frames depicting platinum nanoparticles on a cerium oxide support.

Original: A major challenge for the application of unsupervised metrics is the presence of local correlations in the noise (see Figure 11).
Revised: A major challenge for the application of unsupervised metrics is the presence of local correlations in the noise (see Figure 13).

Original: We address this by performing spatial subsampling to reduce the correlation and selecting two contiguous test sets with low correlation: 155 images with moderate signal-to-noise ratio (SNR), and 383 images with low SNR, which are more challenging.
Revised: We address this by performing spatial subsampling to reduce the correlation and selecting two contiguous test sets with low correlation: 155 images with moderate signal-to-noise ratio (SNR), and 383 images with low SNR, which are more challenging.

Original: We train an unsupervised Neighbor2Neighbor convolutional-neural network Huang et al.
Revised: We train an unsupervised Neighbor2Neighbor convolutional-neural network Huang et al.

Original: (2021) on a training set containing 70% of the data, and compare its performance to a Gaussian-smoothing baseline on the two test sets.
Revised: (2021) on a training set containing 70% of the data, and compare its performance to a Gaussian-smoothing baseline on the two test sets.

Original: Section H provides a more detailed description of the dataset and the models.
Revised: Section J provides a more detailed description of the dataset and the models.

Original: Figure 5 shows examples from the data and the corresponding denoised images, as well as the uPSNR of each method for the two test sets.
Revised: Figure 4 shows examples from the data and the corresponding denoised images, as well as the uPSNR of each method for the two test sets.

Original: Figure 10 shows a histogram of the uMSE values for each individual test image.
Revised: Figure 11 shows a histogram of the uMSE values for each individual test image.

Original: The unsupervised metrics indicate that the deep-learning method achieves effective denoising on the moderate SNR set, clearly outperforming the Gaussian-smoothing baseline, and also that both methods produce significantly worse results for the low-SNR test set.
Revised: The unsupervised metrics indicate that the deep-learning method achieves effective denoising on the moderate SNR set, clearly outperforming the Gaussian-smoothing baseline, and also that both methods produce significantly worse results for the low-SNR test set.

Revised: Figure 12 shows that uMSE produces consistent image-level evaluations for the two denoisers.

Original: These conclusions are consistent with the visual appearance of the images.
Revised: These conclusions are supported by the visual appearance of the images.

Original:  C ONCLUSION A ND O PEN Q UESTIONS
Revised:  C ONCLUSION A ND O PEN Q UESTIONS

Original: In this work we introduce two novel unsupervised metrics computed exclusively from noisy data, which are asymptotically consistent estimators of the corresponding supervised metrics, and yield accurate approximations in practice.
Revised: In this work we introduce two novel unsupervised metrics computed exclusively from noisy data, which are asymptotically consistent estimators of the corresponding supervised metrics, and yield accurate approximations in practice.

Original: These results are limited to denoising under the assumption of independent noise.
Revised: These results are limited to denoising under the assumption of independent noise.

Original: Several important open questions remain: (1) How to address the bias introduced by spatial subsampling and achieve an unbiased approximation to the MSE from a single noisy image.
Revised: Several important open questions remain: (1) How to address the bias introduced by spatial subsampling and achieve an unbiased approximation to the MSE from a single noisy image.

Original: (2) How to extend the approach to noise distributions that are not pixel-wise independent.
Revised: (2) How to account for noise distributions and artifacts which are not pixel-wise independent (see e.g. Prakash et al.

Original: (3) How to obtain unsupervised approximations of perceptual metrics such as SSIM Wang et al.
Revised: (3) How to obtain unsupervised approximations of perceptual metrics such as SSIM Wang et al.

Original: (4) How to perform unsupervised evaluation formore general inverse problems.
Revised: (4) How to perform unsupervised evaluation for general inverse problems, and related applications such as realistic image synthesis Zwicker et al. A C OMPUTATION OF N OISY R EFERENCES Figure 5 illustrates two approaches to compute the noisy references required by the proposed metrics, which are described in Section 3.3.

Original: A C ONFIDENCE I NTERVALS FOR U NCERTAINTY Q UANTIFICATION
Revised: B C ONFIDENCE I NTERVALS FOR U NCERTAINTY Q UANTIFICATION

Original: The uMSE and uPSNR are estimates of the MSE and PSNR computed from noisy data, so they are inherently uncertain.
Revised: The uMSE and uPSNR are estimates of the MSE and PSNR computed from noisy data, so they are inherently uncertain.

Original: We propose to quantify this uncertainty using confidence intervals obtained via bootstrapping.
Revised: We propose to quantify this uncertainty using confidence intervals obtained via bootstrapping.

Original: Algorithm 7 (Bootstrap confidence intervals) .
Revised: Algorithm 7 (Bootstrap confidence intervals) .

Original: We assume access to a noisy input signal y ∈ R n and three noisy references a , b , c ∈ R n .
Revised: We assume access to a noisy input signal y ∈ R n and three noisy references a , b , c ∈ R n .

Original: For 1 ≤ k ≤ K , build an index set B k by sampling n entries from { 1 , 2 , . . . , n } uniformly and independently at random with replacement.
Revised: For 1 ≤ k ≤ K , build an index set B k by sampling n entries from { 1 , 2 , . . . , n } uniformly and independently at random with replacement.

Original: Then setuMSE k := 1 n (cid:88)
Revised: Then set

Original: [Equation]
Revised: [Equation]

Original: To build 1 − α confidence intervals, 0 < α < 1 for the uMSE and uPSNR set
Revised: To build 1 − α confidence intervals, 0 < α < 1 for the uMSE and uPSNR set

Original: [Equation]
Revised: [Equation]

Original: where q uMSE α/and q uMSE1 − α/ 2 are the α/ 2 and 1 − α/ 2 quantiles of the set { uMSE 1 , . .
Revised: where q uMSE α/and q uMSE1 − α/ 2 are the α/ 2 and 1 − α/ 2 quantiles of the set { uMSE 1 , . .

Original: . , uMSE K } , and q uPSNR α/and q uPSNR1 − α/ 2 are the α/ 2 and 1 − α/ 2 quantiles of the set { uPSNR 1 , . .
Revised: . , uMSE K } , and q uPSNR α/and q uPSNR1 − α/ 2 are the α/ 2 and 1 − α/ 2 quantiles of the set { uPSNR 1 , . .

Original: , uPSNR K } .
Revised: , uPSNR K } .

Original: Theorem 6 establishes that the uMSE is asymptotically normal.
Revised: Theorem 6 establishes that the uMSE is asymptotically normal.

Original: In addition, our numerical experiments show that the distribution of the uMSE is well approximated as Gaussian even for relatively small values of n (see Figure 3).
Revised: In addition, our numerical experiments show that the distribution of the uMSE is well approximated as Gaussian even for relatively small values of n (see Figure 2).

Original: As a result, the bootstrap confidence intervals for the uMSE produced by
Revised: As a result, the bootstrap confidence intervals for the uMSE produced by

Original: Algorithm 7 contain the MSE with probability approximately 1 − α (see Section 13.3 in Efron &
Revised: Algorithm 7 contain the MSE with probability approximately 1 − α (see Section 13.3 in Efron &

Original: Tibshirani (1994)).
Revised: Tibshirani (1994)).

Original: This also implies that the PSNR belongs to the bootstrap confidence intervals for the uPSNR with probability approximately 1 − α because the function that maps the uMSE to the uPSNR and the MSE to the PSNR is monotone (see Section 13.6 in Efron & Tibshirani (1994)).
Revised: This also implies that the PSNR belongs to the bootstrap confidence intervals for the uPSNR with probability approximately 1 − α because the function that maps the uMSE to the uPSNR and the MSE to the PSNR is monotone (see Section 13.6 in Efron & Tibshirani (1994)).

Original: Figure 6 shows a numerical verification that the proposed approach yields valid confidence intervals for MSE in the controlled experiments of Section 5, where the ground-truth clean images are known.
Revised: Figure 6 shows a numerical verification that the proposed approach yields valid confidence intervals for MSE in the controlled experiments of Section 5, where the ground-truth clean images are known.

Original: It also shows that the bias introduced by spatial subsampling for natural images (see Section B), shifts the confidence intervals away from the true MSE.[Figure]
Revised: It also shows that the bias introduced by spatial subsampling for natural images (see Section C), shifts the confidence intervals away from the true MSE.

Original: B S PATIAL S UBSAMPLING
Revised: C S PATIAL S UBSAMPLING

Original: In this section, we propose a method to obtain the noisy references required to estimate uMSE and uPSNR.
Revised: In this section, we propose a method to obtain the noisy references required to estimate uMSE and uPSNR.

Original: We focus our discussion on images, but similar ideas can be applied to videos and time-series data.
Revised: We focus our discussion on images, but similar ideas can be applied to videos and time-series data.

Original: In order to simplify the notation, we consider N × N images.
Revised: In order to simplify the notation, we consider N × N images.

Original: The n -dimensional signals in other sections can be interpreted as vectorized versions of these images with n = N 2 .
Revised: The n -dimensional signals in other sections can be interpreted as vectorized versions of these images with n = N 2 .

Original: We assume that we have available a noisy image I of dimensions 2 N × 2 N .
Revised: We assume that we have available a noisy image I of dimensions 2 N × 2 N .

Original: We extract four noisy references from I by spatial subsampling.
Revised: We extract four noisy references from I by spatial subsampling.

Original: The method is inspired by the Neighbor2Neighbor unsupervised denoising method, which uses random subsampling to generate noisy image pairs during training Huang et al.
Revised: The method is inspired by the Neighbor2Neighbor unsupervised denoising method, which uses random subsampling to generate noisy image pairs during training Huang et al.

Original: Figure 7 illustrates the approach.
Revised: Figure 7 illustrates the approach.

Original: Algorithm 8 (Decomposition via spatial subsampling) .
Revised: Algorithm 8 (Decomposition via spatial subsampling) .

Original: Given an image I ∈ R 2 N × 2 N , let
Revised: Given an image I ∈ R 2 N × 2 N , let

Original: S 1 ( i, j ) :
Revised: S 1 ( i, j ) :

Original: = I (2 i − 1 , 2 j − 1) ,
Revised: = I (2 i − 1 , 2 j − 1) ,

Original: S 2 ( i, j ) :
Revised: S 2 ( i, j ) :

Original: = I (2 i, 2 j − 1) ,
Revised: = I (2 i, 2 j − 1) ,

Original: S 3 ( i, j ) :
Revised: S 3 ( i, j ) :

Original: = I (2 i − 1 , 2 j ) ,
Revised: = I (2 i − 1 , 2 j ) ,

Original: S 4 ( i, j ) :
Revised: S 4 ( i, j ) :

Original: = I (2 i, 2 j ) , 1 ≤ i, j ≤ n.
Revised: = I (2 i, 2 j ) , 1 ≤ i, j ≤ n.

Original: The spatial decomposition of I is equal to four sub-images Y , A , B , C ∈ R N × N where Y ( i, j ) ,
Revised: The spatial decomposition of I is equal to four sub-images Y , A , B , C ∈ R N × N where Y ( i, j ) ,

Original: A ( i, j ) , B ( i, j ) , C ( i, j ) are set equal to S 1 ( i, j ) , S 2 ( i, j ) , S 3 ( i, j ) , S 4 ( i, j ) , or to a random permutation of the four values.
Revised: A ( i, j ) , B ( i, j ) , C ( i, j ) are set equal to S 1 ( i, j ) , S 2 ( i, j ) , S 3 ( i, j ) , S 4 ( i, j ) , or to a random permutation of the four values.

Revised: D E FFECT O F S PATIAL S UBSAMPLING ON THE P ROPOSED M ETRICS

Original: Spatial subsampling generates four noisy sub-images that correspond to the noisy input y and the three noisy references a , b and c in Definition 1. 
Revised: Spatial subsampling generates four noisy sub-images that correspond to the noisy input y and the three noisy references a , b and c in Definition 1. In our derivation of the uMSE, we assume that these [Figure]

Original: In our derivation of the uMSE, we assume that these four noisy signals are generated by corrupting the same ground-truth clean signal with independent noise.
Revised: [Figure] The difference is substantially larger in natural images, because they are less smooth with respect to the resolution than the electron-microscopy images. four noisy signals are generated by corrupting the same ground-truth clean signal with independent noise.

Original: Thisassumption holds for the sub-images in Definition 8 if (1) the underlying clean image is smooth, so that adjacent pixels are approximately equal, and (2) the noise is pixel-wise independent.
Revised: This holds for the sub-images in Definition 8 if (1) the underlying clean image is smooth, so that adjacent pixels are approximately equal, and (2) the noise is pixel-wise independent.

Original: Tables 1 and 3, and Figures 4, 6 and 8 show that these assumptions don’t hold completely for natural images, which introduces a bias in the uMSE.
Revised: Tablesand 3, and Figures 3 and 6 show that these assumptions don’t hold completely for natural images, which introduces a bias in the uMSE.

Original: This bias also exists for the electron-microscopy images but it is much smaller, because the images are smoother.
Revised: This bias also exists for the electron-microscopy images but it is much smaller, because the images are smoother with respect to the pixel resolution.

Revised: Figure 8 shows the relative root mean square error (RMSE) between clean copies of images obtained via spatial subsampling following Algorithm 8 for the natural images (left) and electron-microscopy images (right) used for the experiments in Section 5.

Revised: The difference is substantially larger in natural images, because they are less smooth with respect to the pixel resolution than the electron-microscopy images.

Revised: In order to further analyze the effect of spatial subsampling on the proposed metrics, we performed a controlled experiment where we applied different degrees of smoothing (via a Gaussian filter) to a natural image.

Original: C C OMPARISON W ITH A VERAGING -B ASED MSE E STIMATION
Revised: We evaluated the relative RMSE of the corresponding subsampled references. In addition, we fed the smoothed images contaminated by noise into a denoiser and compared the uMSE of the denoised image with its true MSE. The results are shown in Figure 9. We observe that smoothing results in a stark decrease of both the relative RMSE and the uMSE, suggesting that spatial

Revised: [Figure]

Revised: [Figure]

Revised: | u

Revised: M S

Revised: E - M

Revised: S E |

Revised: [Figure]

Revised: Standard deviation of Gaussian smoothing filter

Revised: R e l a t i v e

Revised: R M

Revised: S E b e t w ee n s u b s a m p li n g s 10 order subsampling is effective as long as the underlying image content is sufficiently smooth with respect to the pixel resolution (as supported also by the results on the electron-microscopy data).

Revised: E C OMPARISON W ITH A VERAGING -B ASED MSE E STIMATION

Original: Existing denoising benchmarks containing images corrupted with real noise perform evaluation by computing the MSE or PSNR using an estimate of the clean image obtained by averaging multiple noisy frames Abdelhamed et al.
Revised: Existing denoising benchmarks containing images corrupted with real noise perform evaluation by computing the MSE or PSNR using an estimate of the clean image obtained by averaging multiple noisy frames Abdelhamed et al.

Original: (2018); Plotz & Roth (2017);
Revised: (2018); Plotz & Roth (2017);

Original: Xu et al.
Revised: Xu et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: In this section, we show both theoretically and numerically that this approach produces a poor estimate of the MSE and PSNR, unless the signal-to-noise ratio of the data is very low, or we use a large number of noisy frames.
Revised: In this section, we show both theoretically and numerically that this approach produces a poor estimate of the MSE and PSNR, unless the signal-to-noise ratio of the data is very low, or we use a large number of noisy frames.

Original: The following lemma shows that in contrast to our proposed metric uMSE, the approximation to the
Revised: The following lemma shows that in contrast to our proposed metric uMSE, the approximation to the

Original: MSE obtained via averaging is biased and not consistent , in the sense that it does not converge to the true MSE when the number of pixels tends to infinity.The metric does converge to the MSE as the
Revised: MSE obtained via averaging is biased and not consistent , in the sense that it does not converge to the true MSE when the number of pixels tends to infinity.

Original: [Figure] The difference is substantially larger in natural images, because they have lower resolution and number of noisy images tends to infinity, but this is of little practical significance, since this number cannot be increased arbitrarily in actual applications.
Revised: The metric does converge to the MSE as the number of noisy images tends to infinity, but this is of little practical significance, since this number cannot be increased arbitrarily in actual applications.

Original: Lemma 9 (MSE via averaging) .
Revised: Lemma 9 (MSE via averaging) .

Original: Consider a clean signal x ∈ R n , an estimate f ( y ) ∈ R n (obtained by applying a denoiser f to the data y ∈ R n ), and m noisy references
Revised: Consider a clean signal x ∈ R n , an estimate f ( y ) ∈ R n (obtained by applying a denoiser f to the data y ∈ R n ), and m noisy references

Original: [Equation]
Revised: [Equation]

Original: where ˜ z [ m ] i , 1 ≤ i ≤ n, 1 ≤ j ≤ m , are i.i.d.
Revised: where ˜ z [ m ] i , 1 ≤ i ≤ n, 1 ≤ j ≤ m , are i.i.d.

Original: zero-mean Gaussian random variables with variance σ 2 .
Revised: zero-mean Gaussian random variables with variance σ 2 .

Original: We define the averaging-based MSE as
Revised: We define the averaging-based MSE as

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: The MSE avg is a biased estimator of the true MSE{+t i o n 
+}
Revised: MSEuMSE M S E e s t i m a {+t i o n 
+}

Original: [Equation]
Revised: [Equation]

Original: since its mean equals
Revised: since its mean equals

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the assumptions [Equation] and linearity of expectation,
Revised: By the assumptions, and linearity of expectation,

Original: [Equation]
Revised: [Equation]

Original: [Figure] MSEuMSE M S E e s t i m a t i o n
Revised: [Equation]

Original: [Figure]
Revised: [Equation]

Original: MSE via averaging As established in Section 4, the proposed uMSE metric is an unbiased estimator of the MSE that is consistent as n → ∞ and only requires m := 3 noisy references.
Revised: As established in Section 4, the proposed uMSE metric is an unbiased estimator of the MSE that is consistent as n → ∞ and only requires m := 3 noisy references.

Original: Figure 9 shows a numerical comparison between uMSE and the averaging-based MSE for one of the natural images used in the experiments of Section 5.
Revised: Figure 10 shows a numerical comparison between uMSE and the averaging-based MSE for one of the natural images used in the experiments of Section 5.

Original: We observe that averaging-based MSE requires m := 1510 in order to match the accuracy achieved by the uMSE with only three noisy references.
Revised: We observe that averaging-based MSE requires m := 1510 in order to match the accuracy achieved by the uMSE with only three noisy references.

Revised: [Table]

Revised: The proposed uMSE metric, computed using three noisy references, is very similar to an averaging-based

Revised: PSNR estimate computed from 10 noisy references.

Revised: The metrics are compared on the datasets and denoising methods described in Section I.

Revised: All numbers in the table are scaled by · 10 −

Original: D A DDITIONAL R ESULTS
Revised: [Table] F A DDITIONAL R ESULTS

Original: This section contains additional results, which are not included in the main paper due to space constraints.
Revised: This section contains additional results, which are not included in the main paper due to space constraints.

Original: They include:
Revised: They include:

Original: • Table 3 shows a controlled comparison of MSE and uMSE on clean ground-truth images, and noisy frames for both natural images with additive Gaussian noise and TEM images with Poisson noise.
Revised: • Table 3 shows a controlled comparison of MSE and uMSE on clean ground-truth images, and noisy frames for both natural images with additive Gaussian noise and TEM images with Poisson noise.

Original: • Table 4 shows a comparison between averaging-based MSE and uMSE on RAW videos with real noise described in Sections 6 and G. • Figure 10 Shows the estimates of uMSE evaluated on TEM data for two denoisers, described in Section 7. •Figure 11 shows the empirical correlation of neighboring pixels in the TEM data, necessitating subsampling in order to evaluate uMSE, uPSNR.
Revised: • Table 4 shows a comparison between averaging-based MSE and uMSE on RAW videos with real noise described in Sections 6 and I. • Figure 11 Shows the estimates of uMSE evaluated on TEM data for two denoisers, described in Section 7. •

Original: [Table]
Revised: Figure 12 Shows that uMSE provides a consistent ordering of images that are easier/harder to denoise, across different denoisers

Original: The proposed uMSE metric, computed using three noisy references, is very similar to an averaging-based PSNR estimate computed from 10 noisy references.
Revised: • Figure 13 shows the empirical correlation of neighboring pixels in the TEM data, necessitating subsampling in order to evaluate uMSE, uPSNR.

Original: The metrics are compared on the datasets and denoising methods described in Section G. All numbers in the table are scaled by · 10 −

Original: [Table]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Equation]

Original: E P ROOFS E. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: The following lemma shows that each individual term in the uMSE is unbiased. {+j 
+}
Revised: The following lemma shows that each individual term in the uMSE is unbiased. {+j 
+}

Revised: C o rr ( p i , p j )

Original: Lemma 10 (Proof in Section E.5.2) .
Revised: Lemma 10 (Proof in Section G.5.2) .

Original: If Conditions 1 and 2 in Section 4 hold,
Revised: If Conditions 1 and 2 in Section 4 hold,

Original: [Equation]
Revised: [Equation]

Original: The proof then follows immediately from linearity of expectation,[Equation] n [Equation]
Revised: The proof then follows immediately from linearity of expectation,

Original: (cid:88) i

Original: = (cid:103) uSE i (cid:35) =

Original: n n (cid:88) i =

Original: E (cid:104) (cid:103) uSE i (cid:105)

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: The following lemma bounds the variance of each individual term in the uMSE.
Revised: The following lemma bounds the variance of each individual term in the uMSE.

Original: Lemma 11 (Proof in Section E.5.3) .
Revised: Lemma 11 (Proof in Section G.5.3) .

Original: Under the assumptions of the theorem,
Revised: Under the assumptions of the theorem,

Original: [Equation]
Revised: [Equation]

Original: The proof then follows from the fact that the variance of a sum of independent random variables is equal to the sum of their variances,
Revised: The proof then follows from the fact that the variance of a sum of independent random variables is equal to the sum of their variances,

Original: [Equation]
Revised: [Equation]

Original: The bound immediately implies convergence in mean square as n → ∞ , which in turn implies convergence in probability.
Revised: The bound immediately implies convergence in mean square as n → ∞ , which in turn implies convergence in probability.

Original: E. P ROOF OF C OROLLARY
Revised: G. P ROOF OF C OROLLARY

Original: The uPSNR is a continuous function of the uMSE, which is the same function mapping the MSE to the PSNR.
Revised: The uPSNR is a continuous function of the uMSE, which is the same function mapping the MSE to the PSNR.

Original: The result then follows from Theorem 4 and the continuous mapping theorem.
Revised: The result then follows from Theorem 4 and the continuous mapping theorem.

Original: E. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: To prove Theorem 6, we express the uMSE as a sum of zero-mean random variables,
Revised: To prove Theorem 6, we express the uMSE as a sum of zero-mean random variables,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and apply the following version of the Lyapunov central limit theorem.
Revised: and apply the following version of the Lyapunov central limit theorem.

Original: Theorem 12 (Theorem 9.2 Breiman (1992)) .
Revised: Theorem 12 (Theorem 9.2 Breiman (1992)) .

Original: Let ˜ t i , 1 ≤ i ≤ n , be independent zero-mean random variables with bounded second and third moments, and let
Revised: Let ˜ t i , 1 ≤ i ≤ n , be independent zero-mean random variables with bounded second and third moments, and let

Original: [Equation]
Revised: [Equation]

Original: If the Lyapunov condition
Revised: If the Lyapunov condition

Original: [Equation]
Revised: [Equation]

Original: holds, then the random variable
Revised: holds, then the random variable

Original: [Equation]
Revised: [Equation]

Original: converges in distribution to a standard Gaussian as n → ∞ .
Revised: converges in distribution to a standard Gaussian as n → ∞ .

Original: To complete the proof we show that the random variable
Revised: To complete the proof we show that the random variable

Original: [Equation]
Revised: [Equation]

Original: satisfies the conditions of Theorem 12.
Revised: satisfies the conditions of Theorem 12.

Original: By Lemma 10 its mean is zero.
Revised: By Lemma 10 its mean is zero.

Original: By Lemma 11 its second moment is bounded.
Revised: By Lemma 11 its second moment is bounded.

Original: To control s n , we apply the following auxiliary lemma, which provides a lower bound on the variance of each term in the uMSE.
Revised: To control s n , we apply the following auxiliary lemma, which provides a lower bound on the variance of each term in the uMSE.

Original: Lemma 13.
Revised: Lemma 13.

Original: Under the assumptions of Theorem 6,
Revised: Under the assumptions of Theorem 6,

Original: [Equation]
Revised: [Equation]

Original: where µ [4]
Revised: where µ [4]

Original: i and σ 2 i denote the fourth central moment and the variance of ˜ a i , ˜
Revised: i and σ 2 i denote the fourth central moment and the variance of ˜ a i , ˜

Original: b
Revised: b

Original: i and ˜ c i .
Revised: i and ˜ c i .

Original: The lemma yields a lower bound for s 2 n ,
Revised: The lemma yields a lower bound for s 2 n ,

Original: [Equation]
Revised: [Equation]

Original: The following lemma controls the numerator in the Lyapunov condition, and also shows that the third moment of ˜ t i is bounded.
Revised: The following lemma controls the numerator in the Lyapunov condition, and also shows that the third moment of ˜ t i is bounded.

Original: Lemma 14 (Proof in Section E.5.5) .
Revised: Lemma 14 (Proof in Section G.5.5) .

Original: Under the assumptions of Theorem 6, there exists a numerical positive constant D such that
Revised: Under the assumptions of Theorem 6, there exists a numerical positive constant D such that

Original: [Equation]
Revised: [Equation]

Original: Combining equation 35 and Lemma 14, we obtain
Revised: Combining equation 35 and Lemma 14, we obtain

Original: [Equation]
Revised: [Equation]

Original: which converges to zero as n → ∞ .
Revised: which converges to zero as n → ∞ .

Original: The Lyapunov condition therefore holds and the proof is complete.
Revised: The Lyapunov condition therefore holds and the proof is complete.

Original: E. P ROOF OF AUXILIARY RESULTS E.5. N OTATION
Revised: G. P ROOF OF AUXILIARY RESULTS G.5. N OTATION

Original: To alleviate notation in our proofs, we define the denoising error err i := f ( y ) i − x i and the centered random variables C
Revised: To alleviate notation in our proofs, we define the denoising error err i := f ( y ) i − x i and the centered random variables C

Original: (˜ a i ) :
Revised: (˜ a i ) :

Original: =
Revised: =

Original: ˜ a
Revised: ˜ a

Original: i − x i , C (˜ b i ) :
Revised: i − x i , C (˜ b i ) :

Original: =
Revised: =

Original: ˜ b i − x i and C (˜ c i ) :
Revised: ˜ b i − x i and C (˜ c i ) :

Original: =
Revised: =

Original: ˜ c i − x i , which are independent, have zero mean and satisfy Var [
Revised: ˜ c i − x i , which are independent, have zero mean and satisfy Var [˜ a i ] =

Original: [Equation]
Revised: E (cid:2) ˜ a 2 i (cid:3) =

Original: a i ] 
Revised: Var[˜ b i ] =

Original: [Equation]
Revised: E [˜ b 2 i ] = Var [˜ c i ] = E (cid:2)

Original: E
Revised: ˜ c

Original: [Equation]
Revised: 2 i (cid:3) .

Original: E.5. P ROOF OF L EMMA
Revised: G.5. P ROOF OF L EMMA

Original: By linearity of expectation and the fact that the variance of independent random variables equals the sum of their variances,
Revised: By linearity of expectation and the fact that the variance of independent random variables equals the sum of their variances,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: E.5. P ROOF OF L EMMA
Revised: G.5. P ROOF OF L EMMA

Original: By linearity of expectation, the fact that the variance of independent random variables equals the sum of their variances and the fact that the mean square is an upper bound on the variance,
Revised: By linearity of expectation, the fact that the variance of independent random variables equals the sum of their variances and the fact that the mean square is an upper bound on the variance,

Original: [Equation]
Revised: [Equation]

Original: where we have also used the fact that µ 22 ≤ µ [4] i by Jensen’s inequality
Revised: where we have also used the fact that µ 2 2 ≤ µ [4] i by Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: E.5. P ROOF OF L EMMA
Revised: G.5. P ROOF OF L EMMA

Original: The variance of independent random variables equals the sum of their variances, so
Revised: The variance of independent random variables equals the sum of their variances, so

Original: [Equation]
Revised: [Equation]

Original: and since the mean of ˜ b i − ˜ c
Revised: and since the mean of ˜ b i − ˜ c

Original: i is zero,
Revised: i is zero,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By the definition of variance and linearity of expectation
Revised: By the definition of variance and linearity of expectation,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: E.5. P ROOF OF L EMMA
Revised: G.5. P ROOF OF L EMMA

Original: By linearity of expectation,
Revised: By linearity of expectation,

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: The final bound in equation 68 is obtained by bounding each term in the sum using the assumption that the maximum entrywise denoising error and the central moments of ˜ a i , ˜
Revised: The final bound in equation 68 is obtained by bounding each term in the sum using the assumption that the maximum entrywise denoising error and the central moments of ˜ a i , ˜

Original: b i and ˜ c
Revised: b i and ˜ c

Original: i are bounded.
Revised: i are bounded.

Original: For example,
Revised: For example,

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] by linearity of expectation we have
Revised: Finally, by linearity of expectation we have

Original: [Equation]
Revised: [Equation]

Original: F D ESCRIPTION OF C ONTROLLED E XPERIMENTS
Revised: H D ESCRIPTION OF C ONTROLLED E XPERIMENTS

Original: In this section, we describe the architectures and training procedure for models used in Section 5.
Revised: In this section, we describe the architectures and training procedure for models used in Section 5.

Original: For our experiments with natural images, we use the pre-trained weights released in Zhang et al.
Revised: For our experiments with natural images, we use the pre-trained weights released in Zhang et al.

Original: (2017a) and Mohan et al.
Revised: (2017a) and Mohan et al.

Original: All models are trained on 180 × 180 natural images from the
Revised: All models are trained on 180 × 180 natural images from the

Original: Berkeley Segmentation Dataset (Martin et al., 2001) synthetically corrupted with Gaussian noise with standard deviation uniformly sampled between 0 and 100 .
Revised: Berkeley Segmentation Dataset (Martin et al., 2001) synthetically corrupted with Gaussian noise with standard deviation uniformly sampled between 0 and 100 .

Original: The training set contains 400 images and is augmented via downsampling, random flips, and random rotations of patches in these images (Zhang et al., 2017a; Mohan et al., 2020).
Revised: The training set contains 400 images and is augmented via downsampling, random flips, and random rotations of patches in these images (Zhang et al., 2017a; Mohan et al., 2020).

Original: We use the standard test set containing 68 images for evaluation.
Revised: We use the standard test set containing 68 images for evaluation.

Original: We describe each of the models we use in detail below.
Revised: We describe each of the models we use in detail below.

Original: Bilateral filter OpenCV implementation for the Bilateral filter with a filter diameter ofpixels and σ value =
Revised: Bilateral filter OpenCV implementation for the Bilateral filter with a filter diameter ofpixels and σ value =

Original: σ space = 1 . 2.
Revised: σ space = 1 . 2.

Original: DnCNN .
Revised: DnCNN .

Original: DnCNN Zhang
Revised: DnCNN Zhang

Original: et al.
Revised: et al.

Original: (2017a) consists of 20 convolutional layers, each consisting of 3 × 3 filters and 64 channels, batch normalization (Ioffe & Szegedy, 2015), and a ReLU nonlinearity.
Revised: (2017a) consists of 20 convolutional layers, each consisting of 3 × 3 filters and 64 channels, batch normalization (Ioffe & Szegedy, 2015), and a ReLU nonlinearity.

Original: It has a skip connection from the initial layer to the final layer, which has no nonlinear units.
Revised: It has a skip connection from the initial layer to the final layer, which has no nonlinear units.

Original: We use the pre-trained weights released by the authors.
Revised: We use the pre-trained weights released by the authors.

Original: UNet .
Revised: UNet .

Original: Our UNet model (Ronneberger et al., 2015) has the following layers: (a) conv1 - Takes in input image and maps to 32 channels with 5 × 5 convolutional kernels.
Revised: Our UNet model (Ronneberger et al., 2015) has the following layers: (a) conv1 - Takes in input image and maps to 32 channels with 5 × 5 convolutional kernels.

Original: (b) conv2 - Input: 32 channels.
Revised: (b) conv2 - Input: 32 channels.

Original: Output: 32 channels.
Revised: Output: 32 channels.

Original: 3 × 3 convolutional kernels.
Revised: 3 × 3 convolutional kernels.

Original: (c) conv3 - Input: 32 channels.
Revised: (c) conv3 - Input: 32 channels.

Original: Output: 64 channels.
Revised: Output: 64 channels.

Original: 3 × 3 convolutional kernels with stride 2.
Revised: 3 × 3 convolutional kernels with stride 2.

Original: (d) conv4 - Input: 64 channels.
Revised: (d) conv4 - Input: 64 channels.

Original: Output: 64 channels.
Revised: Output: 64 channels.

Original: 3 × 3 convolutional kernels.
Revised: 3 × 3 convolutional kernels.

Original: (e) conv5 - Input: 64 channels.
Revised: (e) conv5 - Input: 64 channels.

Original: Output: 64 channels.
Revised: Output: 64 channels.

Original: 3 × 3 convolutional kernels with dilation factor of 2.
Revised: 3 × 3 convolutional kernels with dilation factor of 2.

Original: (f) conv6 - Input: 64 channels.
Revised: (f) conv6 - Input: 64 channels.

Original: Output: 64 channels.
Revised: Output: 64 channels.

Original: 3 × 3 convolutional kernels with dilation factor of 4.
Revised: 3 × 3 convolutional kernels with dilation factor of 4.

Original: (g) conv7 - Transpose Convolution layer.
Revised: (g) conv7 - Transpose Convolution layer.

Original: Input: 64 channels.
Revised: Input: 64 channels.

Original: Output: 64 channels.
Revised: Output: 64 channels.

Original: 4 ×filters with stride 2 .
Revised: 4 ×filters with stride 2 .

Original: (h) conv8 - Input: 96 channels.
Revised: (h) conv8 - Input: 96 channels.

Original: Output: 64 channels.
Revised: Output: 64 channels.

Original: 3 × 3 convolutional kernels.
Revised: 3 × 3 convolutional kernels.

Original: The input to this layer is the concatenation of the outputs of layer conv7 and conv2 .
Revised: The input to this layer is the concatenation of the outputs of layer conv7 and conv2 .

Original: (i) conv9 - Input: 32 channels.
Revised: (i) conv9 - Input: 32 channels.

Original: Output: 1 channels.
Revised: Output: 1 channels.

Original: 5 × 5 convolutional kernels.
Revised: 5 × 5 convolutional kernels.

Original: We use pre-trained weights released by the authors of Mohan et al.
Revised: We use pre-trained weights released by the authors of Mohan et al.

Original:  The simplified version of the DenseNet architecture (Huang et al., 2017) hasblocks in total.
Revised: DenseNet The simplified version of the DenseNet architecture (Huang et al., 2017) hasblocks in total.

Original: Each block is a fully convolutional 5 -layer CNN with 3 × 3 filters andchannels in the intermediate layers with ReLU nonlinearity.
Revised: Each block is a fully convolutional 5 -layer CNN with 3 × 3 filters andchannels in the intermediate layers with ReLU nonlinearity.

Original: The first three blocks have an output layer with 64 channels, while the last block has an output layer with only one channel.
Revised: The first three blocks have an output layer with 64 channels, while the last block has an output layer with only one channel.

Original: The output of the i th block is concatenated with the input noisy image and then fed to the ( i + 1) th block, so the last three blocks have 65 input channels.
Revised: The output of the i th block is concatenated with the input noisy image and then fed to the ( i + 1) th block, so the last three blocks have 65 input channels.

Original: We use pre-trained weights released by the authors of Mohan et al.
Revised: We use pre-trained weights released by the authors of Mohan et al.

Original: For our experiments with electron microscopy data, we use the simulated dataset of Pt nanoparticles introduced in Mohan et al.
Revised: For our experiments with electron microscopy data, we use the simulated dataset of Pt nanoparticles introduced in Mohan et al.

Original: Specifically, we used a subset of 5,583 images corresponding to white contrast (the simulated dataset is divided into white, black and intermediate contrast by a domain expert, see Mohan et al.
Revised: Specifically, we used a subset of 5,583 images corresponding to white contrast (the simulated dataset is divided into white, black and intermediate contrast by a domain expert, see Mohan et al.

Original: for more details).
Revised: for more details).

Original: 90% of the data were used for training.
Revised: 90% of the data were used for training.

Original: The remaining 559 images were evenly split into validation and test sets.
Revised: The remaining 559 images were evenly split into validation and test sets.

Original: The UNet architecture used in these experiments is the one introduced in Mohan et al.
Revised: The UNet architecture used in these experiments is the one introduced in Mohan et al.

Original: (2022) with 4 scales and 32 base channels.
Revised: (2022) with 4 scales and 32 base channels.

Original: In addition to bilateral filter, UNet, and DnCNN models described for natural images, we used a blindspot based network.
Revised: In addition to bilateral filter, UNet, and DnCNN models described for natural images, we used a blindspot based network.

Original: BlindSpot Laine et al.
Revised: BlindSpot Laine et al.

Original: (2019) is a CNN which is constrained to predict the intensity of a pixel as a function of the noisy pixels in its neighbourhood, without using the pixel itself.
Revised: (2019) is a CNN which is constrained to predict the intensity of a pixel as a function of the noisy pixels in its neighbourhood, without using the pixel itself.

Original: Following Laine et al.
Revised: Following Laine et al.

Original: Sheth et al.
Revised: Sheth et al.

Original: (2021), we use a UNet architecture as the model backbone.
Revised: (2021), we use a UNet architecture as the model backbone.

Original: G D ESCRIPTION OF E XPERIMENTS WITH V IDEOS IN RAW F ORMAT
Revised: I D ESCRIPTION OF E XPERIMENTS WITH V IDEOS IN RAW F ORMAT

Original: As explained in Section 6, the dataset contains 11 unique videos, each containing 7 frames, captured at five different ISO levels using a surveillance camera.
Revised: As explained in Section 6, the dataset contains 11 unique videos, each containing 7 frames, captured at five different ISO levels using a surveillance camera.

Original: Each video has 10 different noise realizations per frame, which are averaged to obtain an estimated clean version of the video.
Revised: Each video has 10 different noise realizations per frame, which are averaged to obtain an estimated clean version of the video.

Original: Following Sheth et al.
Revised: Following Sheth et al.

Original: (2021), we perform evaluation on five videos from the test test.
Revised: (2021), we perform evaluation on five videos from the test test.

Original: The methods we use:
Revised: The methods we use:

Original: Image Denoiser (Wavelet) .
Revised: Image Denoiser (Wavelet) .

Original: We use Daubechies wavelet to perform denoising, which is the default choice in skimage.restoration , a widely used image restoration package.
Revised: We use Daubechies wavelet to perform denoising, which is the default choice in skimage.restoration , a widely used image restoration package.

Original: We implement denoising using the function denoise_wavelet() from the package using the default options.
Revised: We implement denoising using the function denoise_wavelet() from the package using the default options.

Original: We set sigma=0.01 .
Revised: We set sigma=0.01 .

Original: Image Denoiser (CNN) .
Revised: Image Denoiser (CNN) .

Original: We perform image denoising by re-purposing the video denoiser (UDVD) trained for RAW videos in Ref.
Revised: We perform image denoising by re-purposing the video denoiser (UDVD) trained for RAW videos in Ref.

Original: Sheth et al.
Revised: Sheth et al.

Original: UDVD takes in five consecutive frames, and output the denoised image corresponding to the frame in the middle.
Revised: UDVD takes in five consecutive frames, and output the denoised image corresponding to the frame in the middle.

Original: To simulate image denoising using UDVD, we repeat the same frame 5 times (i.e, all frames are the same image), and provide it as input to the trained network.
Revised: To simulate image denoising using UDVD, we repeat the same frame 5 times (i.e, all frames are the same image), and provide it as input to the trained network.

Original: Video Denoiser (Temp.
Revised: Video Denoiser (Temp.

Original: Avg.) .
Revised: Avg.) .

Original: We use 5 consecutive frames to compute the denosied image corresponding to the middle frame.
Revised: We use 5 consecutive frames to compute the denosied image corresponding to the middle frame.

Original: We assign a weight of 0 .
Revised: We assign a weight of 0 .

Original: 75 to the middle noisy frame, 0 .
Revised: 75 to the middle noisy frame, 0 .

Original: 1 to each of the previous and next frame, and 0 .
Revised: 1 to each of the previous and next frame, and 0 .

Original: 025 to the rest of the two frames.
Revised: 025 to the rest of the two frames.

Original: Video Denoiser (CNN) .
Revised: Video Denoiser (CNN) .

Original: We use the unsupervised video denoiser (UDVD) trained for RAW videos in Ref.
Revised: We use the unsupervised video denoiser (UDVD) trained for RAW videos in Ref.

Original: Sheth et al.
Revised: Sheth et al.

Original: As explained, UDVD takes in five consecutive frames, and output the denoised image corresponding to the frame in the middle.
Revised: As explained, UDVD takes in five consecutive frames, and output the denoised image corresponding to the frame in the middle.

Original: We use the pre-trained weights, and follow the experimental setup described in Ref.
Revised: We use the pre-trained weights, and follow the experimental setup described in Ref.

Original: Sheth et al.
Revised: Sheth et al.

Original: We use the pre-trained weights released by the authors of Sheth et al.
Revised: We use the pre-trained weights released by the authors of Sheth et al.

Original: (2021) as our image and video denoiser.
Revised: (2021) as our image and video denoiser.

Original: These weights are obtained by training UDVD on the first 9 realizations of the 5 videos from the test set of the raw video dataset, holding out the last realization for early stopping (see Sheth et al.
Revised: These weights are obtained by training UDVD on the first 9 realizations of the 5 videos from the test set of the raw video dataset, holding out the last realization for early stopping (see Sheth et al.

Original: (2021) for more details).
Revised: (2021) for more details).

Original: H D ESCRIPTION OF E XPERIMENTS ON E LECTRON M ICROSCOPY D ATA Data acquisition:
Revised: J D ESCRIPTION OF E XPERIMENTS ON E LECTRON M ICROSCOPY D ATA Data acquisition:

Original: The dataset contains TEM images of Pt nanoparticles on a CeO2 substrate.
Revised: The dataset contains TEM images of Pt nanoparticles on a CeO2 substrate.

Original: An electron beam interacts with the sample, and then its intensity is recorded on an imaging plane by the detector.
Revised: An electron beam interacts with the sample, and then its intensity is recorded on an imaging plane by the detector.

Original: The pixel intensity approximately follows a Poisson distribution with parameter equal to the intensity of the electron beam.
Revised: The pixel intensity approximately follows a Poisson distribution with parameter equal to the intensity of the electron beam.

Original: The data were recorded at room temperature at a pressure of ∼ 10 − 6 Torr.
Revised: The data were recorded at room temperature at a pressure of ∼ 10 − 6 Torr.

Original: The electron beam intensity was 600 e/ Å 2 /s .
Revised: The electron beam intensity was 600 e/ Å 2 /s .

Original: The instances are part of 25 videos, taken at a frame rate of 75 frames per second.
Revised: The instances are part of 25 videos, taken at a frame rate of 75 frames per second.

Original: The instances show Pt particles in the size range 1 - 5 nm.
Revised: The instances show Pt particles in the size range 1 - 5 nm.

Original: In a subset of frame series, the particles become unstable and undergo structural dynamic re-arrangements.
Revised: In a subset of frame series, the particles become unstable and undergo structural dynamic re-arrangements.

Original: The periods of instability are punctuated by periods of relative stability.
Revised: The periods of instability are punctuated by periods of relative stability.

Original: Consequently, the nanoparticles show a variety of different sizes and shapes and are also viewed along many different crystallographic directions.
Revised: Consequently, the nanoparticles show a variety of different sizes and shapes and are also viewed along many different crystallographic directions.

Original: Data were collected using a FEI Titan ETEM in EFTEM mode, Gatan Tantalum hot stage, K3 camera in
Revised: Data were collected using a FEI Titan ETEM in EFTEM mode, Gatan Tantalum hot stage, K3 camera in

Original: CDS counting mode.
Revised: CDS counting mode.

Original: Pixel-wise correlation: Our proposed unsupervised metrics rely on the assumption that the noise is pixel-wise independence.
Revised: Pixel-wise correlation: Our proposed unsupervised metrics rely on the assumption that the noise is pixel-wise independence.

Original: This is not the case for this dataset, as shown in Figure 11.We address this by performing spatial subsampling by a factor of two, which reduces the pixel-wise correlation by an order of magnitude.
Revised: This is not the case for this dataset, as shown in Figure 13.We address this by performing spatial subsampling by a factor of two, which reduces the pixel-wise correlation by an order of magnitude.

Original: After this, some frames still present relatively high pixel-wise correlations.
Revised: After this, some frames still present relatively high pixel-wise correlations.

Original: We therefore select the test sets from two sets of contiguous frames with low correlation.
Revised: We therefore select the test sets from two sets of contiguous frames with low correlation.

Original: Training and test sets: The data were divided into three sets: training & validation set, consisting of 70% of the data, and two contiguous test sets with pixel-wise correlation: one containing 155 images with moderate signal-to-noise ratio (SNR), and one containing 383 images with low SNR, which are more challenging.
Revised: Training and test sets: The data were divided into three sets: training & validation set, consisting of 70% of the data, and two contiguous test sets with pixel-wise correlation: one containing 155 images with moderate signal-to-noise ratio (SNR), and one containing 383 images with low SNR, which are more challenging.

Original: The moderate SNR test set is interspersed with the training and validation sets, and contains frames similar to those used to train the network.
Revised: The moderate SNR test set is interspersed with the training and validation sets, and contains frames similar to those used to train the network.

Original: The low SNR test set contains frames which are temporally separated from the training and validation sets, and contains nanoparticle with different structures.
Revised: The low SNR test set contains frames which are temporally separated from the training and validation sets, and contains nanoparticle with different structures.

Original: Denoisers We compare the performance of two denoisers: (1) a convolutional neural network based on Neighbor2Neighbor with the same architecture as in Huang et al.
Revised: Denoisers We compare the performance of two denoisers: (1) a convolutional neural network based on Neighbor2Neighbor with the same architecture as in Huang et al.

Original: (2) Gaussian smoothing with kernel size =25 and σ = 25 .
Revised: (2) Gaussian smoothing with kernel size =25 and σ = 25 .

Original: CNN training parameters
Revised: CNN training parameters

Original: The CNN was based on the architecture in Huang et al.
Revised: The CNN was based on the architecture in Huang et al.

Original: The network consists of a modified UNet, and is trained for 500 epochs using an Adam optimizer with an initial learning rate of 0.0001, and scheduled reduction of the learning rate every 100 epochs.
Revised: The network consists of a modified UNet, and is trained for 500 epochs using an Adam optimizer with an initial learning rate of 0.0001, and scheduled reduction of the learning rate every 100 epochs.

Original: The network has a total of 1,256,689 parameters.
Revised: The network has a total of 1,256,689 parameters.