F ULLY U NSUPERVISED D IVERSITY D ENOISING WITH C ONVOLUTIONAL V ARIATIONAL A UTOENCODERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Learning based methods have emerged as the indisputable leaders for virtually all image restoration tasks.
Especially in the domain of microscopy images, various content-aware image restoration (CARE) approaches are now used to improve the interpretability of acquired data.
Naturally, there are limitations to what can be restored in corrupted images, and like for all inverse problems, many potential solutions exist, and one of them must be chosen.
Here, we propose D IV N OISING , a denoising approach based on fully convolutional variational autoencoders (VAEs), overcoming the problem of having to choose a single solution by predicting a whole distribution of denoised images.
First we introduce a principled way of formulating the unsupervised denoising problem within the VAE framework by explicitly incorporating imaging noise models into the decoder.
Our approach is fully unsupervised, only requiring noisy images and a suitable description of the imaging noise distribution.
We show that such a noise model can either be measured, bootstrapped from noisy data, or even co-learned during training.
If desired, consensus predictions can be inferred from a set of D IV N OISING predictions, leading to competitive results with other unsupervised methods and, on occasion, even with the supervised state-of-the-art.
D IV N OISING samples from the posterior enable a plethora of useful applications.
We are p i q showing denoising results for 13 datasets, p ii q discussing how optical character recognition (OCR) applications can beneﬁt from diverse predictions, and are p iii q demonstrating how instance cell segmentation improves when using diverse D IV N OISING predictions.
I NTRODUCTION
The goal of scientiﬁc image analysis is to analyze pixel-data and measure the properties of objects of interest in images.
Pixel intensities are subject to undesired noise and other distortions, motivating an initial preprocessing step called image restoration .
Image restoration is the task of removing unwanted noise and distortions, giving us clean images that are closer to the true but unknown signal.
In the past years, Deep Learning (DL) has enabled tremendous progress in image restoration (Mao et al., 2016;
Zhang et al., 2017b; Zhang et al., 2017; Weigert et al., 2018).
Supervised DL methods use corresponding pairs of clean and distorted images to learn a mapping between the two quality levels.
The utility of this approach is especially pronounced for microscopy image data of biological samples (Weigert et al., 2017; 2018;
Ouyang et al., 2018; Wang et al., 2019), where quantitative downstream analysis is essential.
More recently, unsupervised content-aware image restoration (CARE) methods (Lehtinen et al., 2018; Krull et al., 2019; Batson & Royer, 2019; Buchholz et al., 2019) have emerged.
They can, enabled by sensible assumptions about the statistics of imaging noise, learn a mapping from noisy to clean images, without ever seeing clean data during training.
Some of these methods additionally include a probabilistic model of the imaging noise (Krull et al., 2020;
Laine et al., 2019; Prakash et al., 2019; Khademi et al., 2020) to further improve their performance.
Note that such denoisers can directly be trained on a given body of noisy images.
All existing approaches have a common ﬂaw: distortions degrade some of the information content in images, generally making it impossible to fully recover the desired clean signal with certainty.
Even an ideal method cannot know which of many possible clean images really has given rise to the degraded observation at hand.
Hence, any restoration method has to make a compromise between possible solutions when predicting a restored image.
Generative models, such as VAEs, are a canonical choice when a distribution over a set of variables needs to be learned.
Still, so far VAEs have been overlooked as a method to solve unsupervised image denoising problems.
This might also be due to the fact that vanilla VAEs (Kingma & Welling, 2014; Rezende et al., 2014) show sub-par performance on denoising problems (see Section 6).
Here we introduce D IV N OISING , a principled approach to incorporate explicit models of the imaging noise distribution in the decoder of a VAE.
Such noise models can be either measured or derived (bootstrapped) from the noisy image data alone (Krull et al., 2020; Prakash et al., 2019).
Additionally we propose a way to co-learn a suitable noise model during training, rendering D IV N OISING fully unsupervised.
We show on 13 datasets that fully convolutional VAEs, trained with our proposed
D IV N OISING framework, yield very competitive results, in 8 cases actually becoming the new stateof-the-art (see Fig.
2 and Table 1).
Still, the key beneﬁt of D IV N OISING is that the method does not need to commit to a single prediction, but is instead capable of generating diverse samples from an approximate posterior of possible true signals.
(Note that point estimates can still be inferred if desired, as shown in Fig.
Other unsupervised denoising methods only provide a single solution (point estimate) of that posterior (Krull et al., 2019; Lehtinen et al., 2018; Batson & Royer, 2019) or predict an independent posterior distribution of intensities per pixel (Krull et al., 2020; Laine et al., 2019;
Prakash et al., 2019; Khademi et al., 2020).
Hence, D IV N OISING is the ﬁrst method that learns to approximate the posterior over meaningful structures in a given body of images.
We believe that D IV N OISING will be hugely beneﬁcial for computational biology applications in biomedical imaging, where noise is typically unavoidable and huge datasets need to be processed on a daily basis.
Here, D IV N OISING enables unsupervised diverse SOTA denoising while requiring only comparatively little computational resources, rendering our approach particularly practical.
Finally, we discuss the utility of diverse denoising results for OCR and showcase it for a ubiquitous analysis task in biology – the instance segmentation of cells in microscopy images (see Fig. 5).
Hence, D IV N OISING has the potential to be useful for many real-world applications and will not only generate state-of-the-art (SOTA) restored images, but also enrich quantitative downstream processing.
R ELATED W ORK
Classical Denoising.
The denoising problem has been addressed by a variety of ﬁltering approaches.
Arguably some of the most prominent ones are Non-Local Means (Buades
et al., 2005) and BM3D (Dabov et al., 2007), which implement a sophisticated non-local ﬁltering scheme.
A comprehensive survey and in-depth discussion of such methods can be found in (Milanfar, 2012).
DL Based Denoising.
Deep Learning methods which directly learn a mapping from noisy image to its clean counterpart (see e.g .
(Zhang et al., 2017a)
and (Weigert et al., 2018)) have outperformed classical denoising methods in recent years.
Two well known contributions are the seminal works by Zhang et al.
(2017a) and later by Weigert et al.
More recently, a number of unsupervised variations have been proposed, and in Section 1 we have described their advantages and disadvantages in detail.
One additional interesting contribution was made by Ulyanov et al.
(2018), introducing a quite different kind of unsupervised restoration approach.
Their method, Deep Image Prior , trains a network separately for each noisy input image in the training set, making this approach rather computationally expensive.
Furthermore, training has to be stopped after a suitable but a priori unknown number of training steps. Recently, Quan et al.
(2020) proposed an interesting method called S ELF 2S ELF which trains a
U-N ET like architecture requiring only single noisy images.
The key idea of this approach is to use blind spot masking, similar to Krull et al.
(2019), together with dropout (Srivastava et al., 2014), which avoids overﬁtting and allows sampling of diverse solutions. Similar to D IV N OISING , the single denoised result is obtained by averaging many diverse predictions. Diverse results obtained via dropout are generally considered to capture the so called epistemic or model uncertainty (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017), i.e . the uncertainty arising from the fact that we have a limited amount of training data available. In contrast, D IV N OISING combines a VAE and a model of the imaging noise to capture what is known as aleatoric or data uncertainty (Böhm et al., 2019; Sensoy et al., 2020), i.e .
the unavoidable uncertainty about the true signal resulting from noisy measurements.
Like in Ulyanov et al.
(2018), also S ELF 2S ELF trains separately on each image that has to be denoised.
While this renders the method universally applicable, it is computationally prohibitive when applied to large datasets, as they are common in microscopy and medical imaging.
The same is true for real time applications such as facial denoising.
D IV N OISING , on the other hand, is trained only once on a given body of data.
Afterwards, it can be efﬁciently applied to new images.
A detailed comparison of S ELF 2S ELF and D IV N OISING in terms of denoising performance, run time and GPU memory requirements can be found in Appendix A.14 and Appendix Table 2.
Denoising (Variational) Autoencoders. Despite the suggestive name, denoising variational autoencoders (Im et al., 2017) are not solving denoising problems.
Instead, this method proposes to add noise to the input data in order to boost the quality of encoder distributions.
This, in turn, can lead to stronger generative models.
Other methods also follow a similar approach to improve overall performance of autoencoders (Vincent et al., 2008; 2010; Jiao et al., 2020).
VAEs for Diverse Solution Sampling.
Although not explored in the context of unsupervised denoising, VAEs are designed to sample diverse solutions from trained posteriors.
The probabilistic
U-N ET (Kohl et al., 2018; 2019) uses conditional VAEs to learn a conditional distribution over segmentations.
Baumgartner et al.
(2019) improve the diversity of segmentation samples by introducing a hierarchy of latent variables to model segmentations at multiple resolutions.
Unlike D IV N OISING , both methods rely on paired training data.
Nazabal et al.
(2020) employ VAEs to learn the distribution of incomplete and heterogeneous data in a fully unsupervised manner.
Babaeizadeh et al.
(2017) build upon a VAE style framework to predict multiple plausible future frames of videos conditioned on given context frames.
A variational inference approach was used by Balakrishnan et al.
(2019) to generate multiple deprojected samples for images and videos collapsed in either spatial or temporal dimensions.
Unlike all these approaches, we address the uncertainty introduced by common imaging noise and show how denoised samples can improve downstream processing.
T HE D ENOISING T ASK
Image restoration is the task of estimating a clean signal s “ p s 1 , . . .
, s N q from a corrupted observation x “ p x 1 , . .
, x N q , where s i and x i , refer to the respective pixel intensities.
The corrupted x is thought to be drawn from a probability distribution p NM
p x | s q , which we call the observation likelihood or the noise model .
In this work we focus on restoring images that suffer from insufﬁcient illumination and detector/camera imperfections.
Contrary to existing methods, D IV N OISING is designed to capture the inherent uncertainty of the denoising problem by learning a suitable posterior distribution.
Formally, the posterior we are interested in is p p s | x q9 p p x | s q p p s q and depends on two components: the prior distribution p p s q of the signal as well as the observation likelihood p NM p x | s q we introduced above.
While the prior is a highly complex distribution, the likelihood p p x | s q of a given imaging system (camera/microscope) can be described analytically (Krull et al., 2020).
Models of Imaging Noise.
The noise model is usually thought to factorize as a product of pixels, implying that the corruption, given the underlying signal, is occurring independently in each pixel as
[Equation]
This assumption is known to hold true for Poisson shot noise and camera readout noise (Zhang et al., 2019;
Krull et al., 2020; Prakash et al., 2019).
We will refer to the probability p NM p
x i | s i q of observing a particular noisy value x i at a pixel i given clean signal s i as the pixel noise model .
Various types of pixel noise models have been proposed, ranging from physics based analytical models (Zhang et al., 2019; Luisier et al., 2010; Foi et al., 2008) to simple histograms (Krull et al., 2020).
In this work, we follow the Gaussian Mixture Model (GMM)
based noise model description of (Prakash et al., 2019).
The parameters of a noise model can be estimated whenever pairs p x 1 , s 1 q of corresponding noisy and clean calibration images are available (Krull et al., 2020).
The signal s 1 « 1 M ř Mj “ 0 x 1 j can then be computed by averaging these noisy observations (Prakash et al., 2019).
In a case where no calibration data can be acquired, s 1 can be estimated by a simple bootstrapping approach (Prakash et al., 2019).
Later we additionally show how a suitable noise model can be co-learned during training.
T HE V ARIATIONAL A UTOENCODER (VAE)
We want to brieﬂy introduce the VAE approach introduced by Kingma & Welling (2014).
A more complete introduction to the topic can be found in (Doersch, 2016; Kingma & Welling, 2019).
VAEs are generative models, capable of learning complex distributions over images x , such as hand written digits (Kingma & Welling, 2014) or faces (Huang et al., 2018).
To achieve this, VAEs use a latent variable z with a ﬁxed (usually a unit normal distribution) prior p p z q and describe
[Equation]
Like conventional autoencoders, they consist of two components: A decoder network g θ p z q that takes a point in latent space and maps it to a distribution p θ p x | z q in image space and an encoder network f φ p x q , which takes an observed image and maps it to a distribution q φ p z | x q in latent space.
By φ and θ , we denote network parameters of the encoder and decoder, respectively.
Note that the decoder alone (together with a suitable prior p p z q ) is sufﬁcient to completely describe the generative model in Eq.
It is usually modelled to factorize over pixels
[Equation]
where p θ p x i | z q is a normal distribution, with its mean and variance predicted by the decoder network network g θ p x q .
The encoder distribution is modelled in a similar fashion, factorizing over the dimensions of the latent space.
Training for VAEs consists of adjusting the parameters θ to make sure that Eq.
2 ﬁts the distribution of training images x .
Kingma et al .
show that this can be achieved with the help of the encoder by jointly optimizing φ and θ to minimize the loss L φ,θ p x q “ L R φ,θ p x q ` L KL φ p x q , where
L
[Equation]
ı , and L KL φ p x q is the KL divergence KL p
q φ p z | x q|| p p z qq .
While L KL φ p x q can be computed analytically, the expected value in
L R φ,θ p x q is approximated by drawing a single sample z 1 from q φ p z | x q and using the reparametrization trick by Kingma & Welling (2014) for gradient computation.
D IV N OISING
In D IV N OISING , we build on the VAE setup but interpret it from a denoising-speciﬁc perspective.
We assume that images have been created from a clean signal s via a known noise model, i.e., x „ p NM p x | s q .
To account for this within the VAE setup, we replace the generic normal distribution over pixel intensities in Eq.
3 with a known noise model p NM p
x
| s q (see Eq. 1).
We get p θ p x | z q “ p NM p x | s q “ ś Ni p NM p x
i | s i q , with the decoder now predicting the signal g θ p z q “ s .
Together with p p z q and the noise model, the decoder now describes a full joint model for all three variables, including the signal:
[Equation]
where we assume that p NM p x | s , z q “ p NM p x | s q .
For a given z k , as for standard VAEs, the decoder describes a distribution over noisy images p p x | z q .
The corresponding clean signal s k , in contrast, is deterministically deﬁned.
Hence, p θ p s | z q is a Dirac distribution centered at g θ p z q .
Training.
Considering
Eq.
1, the reconstruction loss becomes
[Equation]
ı .
Apart from this modiﬁcation, we can follow the standard VAE training procedure, just as described in Section 4.
Since we have only modiﬁed how the decoder distribution is modeled, we can assume that the training procedure still produces p i q a model describing the distribution of our training data, while p ii q making sure that the encoder distribution well approximates the distribution of the latent variable given the image.
A complete derivation of the D IV N OISING loss (from probability model perspective) can be found in Appendix A.12.
Prediction.
While we can use the trained VAE to generate images from p θ p x q (see Appendix A.5), here we are mainly interested in denoising.
Hence, we desire access to
the posterior p p
s | x q , i.e . the distribution of possible clean signals s given a noisy observation x .
Assuming the encoder and decoder are sufﬁciently well trained, samples s k from an approximate posterior can be obtained by p
i q feeding the noisy image x into our encoder, p ii q drawing samples
z k „ q φ p z | x q , and p iii q decoding the samples via the decoder to get s k “ g θ p z k q .
Inference.
Given a set of posterior samples s k for a noisy image x , we can infer different consensus estimates (point estimates).
We can, for example, approximate the MMSE estimate (see Fig. 2), by averaging many samples s k .
Alternatively, we can attempt to ﬁnd the maximum a posteriori (MAP) estimate, i.e .
the most likely signal given the noisy observation x , by ﬁnding the mode of the [Figure] [Table] [Figure]
[Table] [Table] posterior distribution. For this purpose, we iteratively use the mean shift algorithm (Cheng, 1995) with decreasing bandwidth to ﬁnd the mode of our sample set (see Fig.
4 and Appendix A.4).
Fully Unsupervised DivNoising.
So far we explained our setup under the assumption that the noise model can either be measured with paired calibration images, or bootstrapped from noisy data (Prakash et al., 2019).
Here, we propose yet another alternative approach of co-learning the noise model directly from noisy data during training.
More concretely, this is enabled by a simple modiﬁcation to the D IV N OISING decoder.
We assume that the noise at each pixel i follows a normal distribution with its variance being a linear function of s i , i.e ., σ 2 i “ a s i ` b .
Linearity is motivated by noise properties in low-light settings (Faraji & MacLean, 2006; Jezierska et al., 2011).
The learnable network parameters a and b are co-optimized during training.
Since variances cannot be negative, we additionally constrain the predicted values for σ 2 i to be positive (see Appendix A.3 for details).
Denoising with Vanilla VAEs.
While not originally intended for denoising tasks, we were curious to see how vanilla VAEs perform when applied to these problems.
Just like fully unsupervised
D IV N OISING , also the vanilla VAE does not require a noise model.
It does, instead, directly predict per-pixel mean and variance (see Section 4), leaving the possibility open that the right values could be learned.
However, here the decoder is not restricted to make each pixel’s variance a function of predicted signal.
We investigate the denoising performance of the vanilla VAE in Section 6 and show in Fig.3 that the predicted variances signiﬁcantly diverge from ground truth noise distributions. Signal prior in D IV N OISING .
Classical denoising methods often explicitly model the image/signal prior p p s q e.g .
as smoothness priors (Grimson & Grimson, 1981; Li, 1994), sparseness priors (Tibshi- rani, 1996), non-local similarity priors (Buades et al., 2005; Dabov et al., 2007) etc., assuming speciﬁc properties of the images at hand.
They effectively assign the same probability to all images/signals sharing e.g .
the same level of smoothness.
However, the true distribution p p s q of clean signals ( e.g . for a particular experimental setup in a ﬂuorescence microscope) is generally more complex. Instead of explicitly modelling p p s q , D IV N OISING only implicitly describes p θ p s q “ ş p θ p s | z q p p z q d z as integral over all possible values of z . We recall that the prior p p z q is assumed to be the unit Gaussian distribution and the conditional distribution p θ p s | z q is learned by the decoder network as the Dirac distribution centered at g θ p z q .
Depending on its parameters θ , the network will implement the function differently, leading to a different p θ p s | z q , and ultimately to a different p θ p s q .
This implicit distribution is quite powerful and can capture complex structures.
See Appendix A.5 for samples obtained from this signal prior for different datasets.
D ATA , E XPERIMENTS , R ESULTS
We quantitatively evaluated the performance of D IV N OISING on 13 publicly available datasets (see
Appendices A.1 and A.2 for data details), 9 of which are subject to high levels of intrinsic noise.
Toothers we synthetically added noise, hence giving us full knowledge about the nature of added noise.
Denoising Baselines.
We choose state-of-the-art baseline methods to compare against D IV N OISING , namely, the supervised CARE
(Weigert et al., 2018) and the unsupervised methods N OISE 2V OID (N2V) (Krull et al., 2019) and Probabilistic N OISE 2V OID (PN2V) (Krull et al., 2019).
All baselines use the available implementations of (Krull et al., 2020) and, as long as not speciﬁed otherwise, make use of a depth 3 U-N ET with 1 input channel and 64 channels in the ﬁrst layer.
As an additional baseline, we choose vanilla VAE with the same network architecture as D IV N OISING but predicting per pixel mean and variance independently.
Training is performed using the ADAM (Kingma & Ba, 2015) optimizer for 200 epochs with 10 steps per epoch with a batch size of 4 and a virtual batch size of 20 for N2V and CARE and a batch size of 1 and a virtual batch size of 20 for PN2V, an initial learning rate of 0 . 001 , and the same basic learning rate scheduler as in (Krull et al., 2020).
All baselines use on the ﬂy data augmentation (ﬂipping and rotation) during training.
Training details.
In all experiments we use rather small, fully convolutional VAE networks, with either 200 k or 713 k parameters (see Appendix A.3).
For all experiments on intrinsically noisy microscopy data, validation and test set splits follow the ones described in the respective publication.
In contrast to the synthetically noisy data, no apriori noise model is known for microscopy datasets.
For these datasets, we used GMM-based noise models (Prakash et al., 2019; Khademi et al., 2020), which are measured from calibration images, as well as co-learned noise models.
For the W2S datasets, no dedicated calibration samples to create noise models are available.
Hence, for this dataset, we use the available clean ground truth images and all noisy observations of the training data to learn a GMM-based noise model.
All GMM noise models use 3 Gaussians and 2 coefﬁcients each.
Find more training details in Appendix A.3.
Denoising Results.
In Table 1, we report denoising performance of all experiments we conducted in terms of peak signal-to-noise ratio (PSNR) with respect to available ground truth images.
The
D IV N OISING results (using the MMSE estimate from 1000 averaged samples) are typically either on par or even beyond the denoising quality reached by the baselines in the ’fully unsupervised’ category, as well as the ’unsupervised with noise model’ category. Note that sampling is very efﬁcient.
For all presented experiments sampling 1000 images consistently took less than 7 seconds (see Table 3 in
Appendix A.15 for precise sampling times).
The effect of averaging a different number of samples is explored in Appendix A.8.
D IV N OISING MMSE is typically, as expected, slightly behind the performance of the fully supervised baseline CARE
(Weigert et al., 2018).
Additionally, on FU-PN2V Convallaria we have demonstrated that a suitable noise model for D IV N OISING can be created via bootstrapping (Prakash et al., 2019;
Khademi et al., 2020).
We also compare against Deep Image Prior (DIP) on DenoiSeg Flywing dataset as it has smallest number of test images and DIP has to be trained for each image.
DIP achieves PSNR of 24 .
67 ˘ 0 .
050 dB compared to 25 .
02 ˘ 0 .
024 dB with D IV N OISING .
Due to the extensive computational requirements of S ELF 2S ELF , we cannot run the method on all images in any of our dataset.
Instead, we run it on single, randomly selected images from the FUPN2V Convallaria , FU-PN2V Mouse actin , FU-PN2V Mouse nuclei , and W2S Ch.1 (avg1) datasets. We compare S ELF 2S ELF to D IV N OISING when p i q trained on the same randomly chosen image [Figure] [Figure]
from the respective dataset, and p ii q when D IV N OISING was trained on the entire dataset and applied on the respective randomly selected image. Within a generous time limit of 10 hours for training per image, D IV N OISING still outperforms S ELF 2S ELF in measured PSNR performance while requiring about 7 times less GPU memory (see Appendix A.14 and Appendix Table 2). Note that the application of S ELF 2S ELF to an entire dataset containing 100 images would require 1000 hours of cumulative training time, while an overall 10 hour training of D IV N OISING on the entire dataset is sufﬁcient to denoise all contained images. The performance on the natural image benchmark dataset
{+BSD
 (Roth & Black, 2005) is shown in Fig. 25 and discussed+} in Appendix A.10.
Additional qualitative results for all datasets can be found in Appendix A.9. A discussion on the accuracy of the posterior modeled by D IV N OISING can be found in Appendix A.11.
Downstream Processing: OCR.
In Fig.
4 we show how Optical Character Recognition (OCR) applications might beneﬁt from diverse denoising.
While regular denoising approaches predict poor compromises that would never be seen in clean text, D IV N OISING can generate a diverse set of rather plausible denoised solutions.
While our MAP estimates clean up most such problems, occasional mistakes cannot be avoided, e.g . changing "hunger" to "hungor" (see Fig. 4).
Diverse denoising solutions obtained by clustering typically correspond to plausible alternative interpretations.
It stands to reason that OCR systems can beneﬁt from having access to diverse interpretations.
Downstream Processing: Instance Cell Segmentation.
We demonstrate how diverse denoised images generated with D IV N OISING can help to segment all cells in the DenoiSeg Flywing data.
While methods to generate diverse segmentations do exist (Kohl et al., 2018; 2019), they require ground truth segmentation labels during training.
In contrast, we use a simple and fast downstream segmentation pipeline c
p x q based on local thresholding and skeletonization (see Appendix A. for details) and apply it to individual samples ( s 1 . . .
s K ) predicted by D IV N OISING to derive segmentations ( c 1 . . .
c K ).
We explore two label fusion methods to combine the individual results and obtain an improved segmentation.
We do: p i q use Consensus (BIC) (Emre Akbas et al., 2018) and p ii q create a pixel-wise average of ( c 1 . . .
c K ), followed by again applying our threshold based segmentation procedure on this average, calling it Consensus (Avg) .
For comparison, we also segment p i q the low SNR input images, p ii q the original high SNR microscopy images, and p iii q the MMSE solutions of D IV N OISING .
Figure 5 shows all results of our instance segmentation experiments.
It is important to note that segmentation from even a single
D IV N OISING prediction outperforms segmentations on the low SNR image data by a large margin.
We observe that label fusion methods can, by utilizing multiple samples, outperform the MMSE estimate, with Consensus (Avg) giving the best overall results.
D ISCUSSION AND C ONCLUSION
We have introduced D IV N OISING , a novel unsupervised denoising paradigm that allows us, for the ﬁrst time, to generate diverse and plausible denoising solutions, sampled from a learned posterior.
We have demonstrated that the quality of denoised images is highly competitive, typically outperforming the unsupervised state-of-the-art, and at times even improving on supervised results.
D IV N OISING uses a lightweight fully convolutional architecture. The success of Deep Image Prior (Ulyanov et al., 2018) shows that convolutional neural networks are inherently suitable for image denoising. Yokota et al.
(2019) reinforce this idea and Tachella et al.
(2020) additionally hypothesize that a possible reason for the success of convolutional networks is their similarity to non-local patch based ﬁltering techniques.
However, the overall performance of D IV N OISING is not merely a consequence of its convolutional architecture.
We believe that the novel and explicit modeling of imaging noise in the decoder plays an essential role.
This becomes evident when comparing our results to other convolutional baselines (including Deep Image Prior and fully convolutional VAEs), which do not perform as well as D IV N OISING on any of the datasets we used.
Additionally, we observe that incorrect noise models consistently lead to inferior results (see Appendix A.7).
We ﬁnd that D IV N OISING is suited particularly well for microscopy data or other applications on a limited image domain.
In its current form it works less well on collections of natural images (see Appendix A.10).
This might not be very surprising, as we are training a generative model for our image data and would not expect to be capturing the tremendously diverse domain of natural photographic images with the comparatively tiny networks used in our experiments (see
Appendix A.3).
For microscopy data, instead, the diversity between datasets can be huge.
Images of the same type of sample, acquired using the same experimental setup, however, contain many resembling structures of lesser overall diversity (they are from a limited image domain).
Nevertheless, the stunning results we achieve suggest that D IV N OISING will also ﬁnd application in other areas where low SNR limited domain image data has to be analyzed.
Next to microscopy, we can think of astronomy, medical imaging, or limited domain natural images such as faces or street scenes.
Additionally, follow up research will explore larger and improved network architectures, able to capture more complex D IV N OISING posteriors on datasets covering larger image domains.
While we constrained ourselves to the standard per-pixel noise models in this paper, the D IV N OISING approach could in principle also work with more sophisticated higher level image degradation models, as long as they can be probabilistically described.
This might include diffraction, blur, or even compression and demosaicing artefacts.
Maybe most importantly, D IV N OISING can not only produce competitive and diverse results, but these results can also be leveraged for downstream processing.
We have seen that cell segmentation can be improved and that clustering our results provides us with meaningful alternative interpretations of the same data (see Fig. 4).
We believe that this is a highly promising direction for many applications, as it provides us with a way to account for the uncertainty introduced by the imaging process.
We are looking forward to see how D IV N OISING will be applied and extended by the community, showcasing the true potential and limitations of this approach.
A A PPENDIX A. I NTRINSICALLY N OISY M ICROSCOPY D ATA
We use public microscopy datasets which show realistic levels of noise, introduced by the respective optical imaging setups.
The FU-PN2V Convallaria (Krull et al., 2020; Prakash et al., 2019) data, consists of 100 noisy calibration images (intended to generate a noise model), and 100 images of size 1024 ˆ 1024 showing a noisy Convallaria section.
The FU-PN2V Mouse nuclei (Prakash et al., 2019) data is composed of 500 noisy calibration images and 200 noisy images of size 512 ˆshowing labeled cell nuclei.
The FU-PN2V Mouse actin (Prakash et al., 2019) data from the same source consists of 100 noisy calibration images and 100 noisy images of size 1024 ˆ 1024 of the same sample, but labeled for the protein actin.
Finally, we use all 3 channels of 2 noise levels (avg1 and avg16) of the W2S
(Zhou et al., 2020) data.
For each channel, corresponding high quality (ground truth) images are available.
Each channel’s training and test sets consist of 80 and 40 images, respectively.
All images are 512 ˆ 512 pixels in size.
A. D ATA E XPOSED TO S YNTHETIC N OISE
We use the well known MNIST (LeCun et al., 1998) as well as the KMNIST (Clanuwat et al., 2018) dataset showing 28 ˆ 28 images of handwritten digits and phonetic letters of hiragana, respectively.
Both datasets contain 60000 training examples and 10000 test examples.
Onto both datasets we added pixel-wise independent Gaussian noise with µ “ 0 and σ “ 140 .
As a third text-based dataset we rendered the freely available eBook “The Beetle” (Marsh, 2004) and extracted 40800 image patches of size 128 ˆ 128 .
We separated 34680 patches for training and 6120 patches for validation, and added pixel-wise independent Gaussian noise with µ “ 0 and σ “ 255 .
Additionally, we use three datasets from microscopy.
The DenoiSeg Mouse (Buchholz et al., 2020) data, showing cell nuclei in the developing mouse skull, consists of 908 training and 160 validation images of size 128 ˆ 128 , with additional 67 images of size 256 ˆ 256 for testing.
Two noisy datasets were created with this data, one by exposing all images to pixel-wise independent Gaussian noise with µ “ 0 and σ “and another one by ﬁrst applying poisson noise with λ “ 1 followed by adding gaussian noise with µ “ 0 and σ “ 10 followed by randomly changing 3% of pixels to either 0 or 255 .
This dataset is called Mouse s & p in Table 1.
The DenoiSeg Flywing (Buchholz et al., 2020) data is showing membrane labeled cells in a ﬂy wing, consisting of 1428 training and 252 validation patches of size 128 ˆ 128 , with additional 42 images of size 512 ˆ 512 for testing.
We exposed this data to pixel-wise independent Gaussian noise with µ “ 0 and σ “ 70 to create a synthetic low SNR version.
All original datasets are 8-bit.
Lastly, we randomly select 500 images of size 384 ˆ 286 from BioID Face recognition database (noa) and corrupt them with pixel-wise independent Gaussian noise with µ “and σ “ 15 .
We use 340 , 60 and 100 images for training, validation and test respectively.
A. T RAINING AND N ETWORK D ETAILS
Here, we provide additional details about the network architecture and training parameters used throughout the main manuscript.
For all D IV N OISING experiments, we use rather lightweight depth 2 and depth 3 VAE architectures (see Appendix Figs.
6 and 7, respectively).
All networks use a single input channel and 32 feature channels in the ﬁrst network layer except for the network trained on mouse s & p dataset which uses 96 feature channels in the ﬁrst network layer.
We use two 3 ˆ 3 convolutions (with padding 1 ), each followed by ReLU activation, followed by a 2 ˆ 2 max pooling layer.
After each such downsampling step, we double the number of feature channels.
For all experiments we use a network architecture of depth 2 (with 2 down/upsampling steps).
The only exceptions are our experiments on DenoiSeg Flywing and eBook data, for which we use a deptharchitecture (with 3 down/upsampling steps).
In total, our depth 2 networks have only around 200 k parameters and depth 3 networks have around 700 k parameters.
While we generally use a VAE bottleneck of 64 latent space feature dimensions for each pixel of the image (after encoding), for the small 28 ˆ 28 MNIST and KMNIST images we use only 8 such latent space dimensions.
We consistently use 8 -fold data augmentation (rotation and ﬂipping) in all experiments.
All networks are trained with a batch size of 32 and an initial learning rate of 0 .
The learning rate is multiplied by 0 .
5 if the validation loss does not decrease for 30 epochs.
For all datasets other than MNIST and KMNIST, we extract training patches of size 128 ˆ 128 , and separate 15% of all patches for validation.
We set the maximum number of epochs such that approximately 22 million steps are performed, and in each epoch the entire training data is being fed.
Training is terminated if the validation loss does not decrease by at least 10 ´ 6 over 100 epochs.
For DenoiSeg Flywing we observed KL vanishing and solved it via Annealing within the ﬁrstepochs (Bowman et al., 2015).
The fully unsupervised D IV N OISING decoder directly predicts the signal and the noise variance per pixel where the variance is constrained to linearly depend on the signal (See Section 5).
To avoid numerical problems and ensure that the predicted variance always remains positive, we allow the user to set a minimum allowed variance/standard deviation σ 2 min / σ min , and enforce this by clamping the predicted values.
Note that a viable choice for this parameter depends on the intensity range of the dataset.
We use the following values: For all FU-PN2V datasets σ min “ 50 , for DenoiSeg Flywing and DenoiSeg Mouse datasets σ min “ 3 , for DenoiSeg Mouse s & p dataset σ min “ 1 , for BioID Face dataset σ min “ 15 , for W2S avg 1 datasets σ min “ 25 and for W2S avg 16 datasets σ min “ 3 .
Run Time and Hardware Requirements.
D IV N OISING using light weight fully convolutional networks (see Appendix Figs.
6 and 7) runs on relatively cheap computational budget.
Our depth 2 networks trained for all experiments requires about 1 .
8 GB GPU memory and our depth 3 networks roughly 5 GB GPU memory on a NVIDIA
TITAN
Xp GPU.
The training time varied from 5 ´ 12 hours on average depending on the dataset.
Input (1, 128, 128)
Convolution (3, 3) (32, 128, 128) Convolution (3, 3) (32, 128, 128)
Maxpooling (2, 2) (32, 64, 64)
Convolution (3, 3) (64, 64, 64) Convolution (3, 3) (64, 64, 64)
Maxpooling (2, 2) (64, 32, 32)
Convolution (3, 3) (64, 32, 32)
Convolution (3, 3) (64, 32, 32) z (64, 32, 32)
Convolution (3, 3) (64, 32, 32) Convolution (3, 3) (64, 32, 32)
Transposed Convolution (3, 3) (64, 64, 64)
Convolution (3, 3) (32, 64, 64) Convolution (3, 3) (32, 64, 64)
Transposed Convolution (3, 3) (32, 128, 128)
Convolution (3, 3) (1, 128, 128)
[Figure]
Reparametrization
[Figure]
Input (1, 128, 128)
Convolution (3, 3) (32, 128, 128) Convolution (3, 3) (32, 128, 128)
Maxpooling (2, 2) (32, 64, 64)
Convolution (3, 3) (64, 64, 64) Convolution (3, 3) (64, 64, 64)
Maxpooling (2, 2) (64, 32, 32)
Convolution (3, 3) (64, 16, 16) Convolution (3, 3) (64, 16, 16) z (64, 16, 16)
Convolution (3, 3) (128,16, 16) Convolution (3, 3) (128, 16, 16)
Transposed Convolution (3, 3) (128, 32, 32)
Convolution (3, 3) (64, 32, 32) Convolution (3, 3) (64, 32, 32)
Transposed Convolution (3, 3) (64, 64, 64)
Convolution (3, 3) (1, 128, 128) µ σ
Reparametrization trick
Convolution (3, 3) (128, 32, 32) Convolution (3, 3) (128, 32, 32)
Maxpooling (2, 2) (128, 16, 16)
Convolution (3, 3) (32, 64, 64) Convolution (3, 3) (32, 64, 64)
Transposed Convolution (3, 3) (32, 128, 128)
A. C LUSTERING OF S OLUTIONS AND D ERIVING THE MAP E STIMATE
Here we provide additional details on how the cluster centers and the approximate MAP estimate of Fig.
4 (see main text) were found.
We ﬁrst drew 10000 sampled images from the approximate posterior as described in Section 4 of the main text.
We then performed mean shift (Cheng, 1995) clustering (using the existing scipy implementation) on the cropped image region shown in the ﬁgure.
We set a bandwidth of 800 and the the maximum number of iterations to 20 , and used the 100 ﬁrst samples of D IV N OISING as seeds.
We ﬁnally show 9 of the resulting cluster centers in the ﬁgure.
To produce the MAP estimate, we employ a similar strategy.
In order to ﬁnd the mode of the sampled distribution efﬁciently, we assume that dependencies in the predicted samples should be local.
This assumption is valid, since our network only has only a ﬁnite receptive ﬁeld for each predicted pixel.
Hence, we apply mean shift algorithm on locally overlapping regions.
We use a window size of 10 ˆ 10 pixels with an overlap of 3 pixels in x and y .
On each such region, the mean shift algorithm is executed repeatedly with decreasing bandwidth, always using the latest result as new seed.
We start by using the sample mean as seed and with an initial bandwidth of 200 .
After each iteration the bandwidth is decreased by a factor of 0 . 9 , until it drops below 100 .
Similar results should also be achievable by applying mean shift algorithm on the entire image.
But since samples will differ at any location in the image, this global approach would require an excessively large number of D IV N OISING samples.
A. G ENERATING I MAGES WITH D IV N OISING M ODELS
Just as with a vanilla VAE (see Section 4 in the main text), we can use a trained D IV N OISING VAE to synthesise images of structures resembling the training data.
To achieve this, we sample from the normal distribution z k „ p p z q and process each sample with the decoder network s k “ g θ p z k q .
We show such generated images in comparison to real crops from the test data in Appendix Figs.
8 to 10.
We see that the images appear most plausible for local structures, indicating that the small networks we use in this work are not capable of capturing larger structural features in the given data.
A. I NSTANCE C ELL S EGMENTATION
Here, we provide additional details regarding the downstream segmentation task described in Sectionof the main text.
We used the ﬁrst 21 images in the test set of DenoiSeg Flywing for our analysis.
Given an input image, our segmentation pipeline consists of p i q generating segmentation masks using local thresholding with a mean ﬁlter of radius 15 , followed by p ii q skeletonizing the space between these masks, followed by p iii q connected component analysis to obtain instance segmentation.
Using this pipeline, we generated segmentation for the noisy (low SNR) images, ground truth (high
SNR) images, as well as for the D IV N OISING MMSE estimate (obtained by averaging 1000 sampled denoised images).
We also apply the above described pipeline for each of the 1000 D IV N OISING samples separately to serve as input for the two label fusion methods, namely p i q Consensus (BIC) , and p ii q Consensus (Avg) .
For the latter label fusion method we skip the connected component analysis and directly average the thresholded and skeletonized images.
To obtain the ﬁnal result, we again apply the full segmentation pipeline described above to this average image.
All segmentations were obtained with the open source image analysis software Fiji (Schindelin et al., 2012).
A. T HE R ELATIVE I MPORTANCE OF THE KL L OSS C OMPONENT
We can generalize our D IV N OISING training loss as a weighted combination of a modiﬁed reconstruction loss (see Section 5 in the main text) and KL divergence loss, where the two loss components are weighted equally.
Following the exposition in (Higgins et al., 2017), we explore the effect of weighting the KL loss component during training with a factor β .
Our modiﬁed training loss thus becomes
[Equation]
(5) where setting β “ 1 gives our D IV N OISING setup described in Section 5 in the main text.
Note that increasing or reducing β , i.e . changing the relative importance of the reconstruction loss, is equivalent to using a wider or narrower noise model, such as a Gaussian noise model with larger or smaller standard deviation σ .
We can thus interpret above results as the effect of using a mismatched noise model that is either too wide or too narrow.
Effect of β on Denoising Quality.
We investigated the effect of β on the denoising ability of
D IV N OISING network with the DenoiSeg Flywing dataset.
As illustrated in Appendix Fig.
11a, β “ 1 gives the optimal results for the MMSE estimate (obtained by averaging 1000 samples).
Both regimes, β ą
and β ă 1 , yield sub-par denoising performance.
Effect of β on Diversity of Denoised Samples.
We introduce a simple new metric, called standard deviation PSNR , to quantify the diversity of denoised results obtained as a function of β .
For a given noisy image x and given a set of denoised samples S x , we compute the PSNR of each sample a P S x with respect to the corresponding ground truth image s .
This yields a vector of PSNR values v where v i “ PSNR p s, a i q , for v i P v .
Standard deviation PSNR for the noisy image x is then deﬁned as the standard deviation of elements in the vector v .
Appendix Fig.
11b reports the average of standard deviation PSNR obtained for 42 test images of the DenoiSeg Flywing dataset.
The higher the beta, the higher is the standard deviation PSNR indicating higher diversity.
Qualitative results presented in
Appendix Fig.
12 show that with β ą 1 , there is an increased diversity at the bigger image scales (e.g. diverse predictions of cell membranes), and generated denoised images appear smoother than those observed in real data.
Setting β
ă 1 reduces diversity and introduces grainy artefacts, thereby yielding poor reconstructions.
Note that β “ 1 gives the best results in terms of PSNR of MMSE while maintaining a fair level of diversity.
A. H OW D OES N OISE A FFECT THE D IVERSITY OF D IV N OISING S AMPLES ?
We quantiﬁed how the diversity of D IV N OISING samples changes with the amount of noise present in the original dataset.
Increased level of noise introduces additional uncertainty about the true signal, hence we would expect this to lead to increasingly diverse samples.
To test this hypothesis
[Figure]
we choose the DenoiSeg Flywing dataset and inject pixel wise independent gaussian noise of mean 0 and standard deviations σ “ 30 , 50 and 70 .
We report the standard deviation PSNR diversity metric, introduced in Appendix Section A.
[Figure]
for all three noise levels.
As demonstrated in Appendix Fig.
11c, the higher the noise level, the more diverse the D IV N OISING samples become, thereby conﬁrming our hypothesis.
[Figure]
[Figure]
[Figure]
averaged v a l u e ( r e l a ti v e i m po r t a n ce o f K
L l o ss )
PS N R ( d
B )     a)
[Figure]
) c)
We also investigate the effect of weighting the KL loss term with a factor β (Supplementary Eq
5) on the quality of reconstruction.
We observe that the usual VAE setup with β “ 1 gives the best results in terms of reconstruction quality.
Increasing β ą 1 leads to higher diversity at the expense of poor reconstruction (see Appendix Fig.
) We quantify the denoising diversity achieved with different in terms of
(see Appendix section A on the investigate
[Figure]
Sample
Sample 1 - Sample 2 Sample 1 - Sample 3 Sample 2 - Sample
MMSE ( β = 4)
[Figure]
β =
[Figure]
β =
[Figure]
MMSE ( β = 2) MMSE ( β = 1)
MMSE ( β = 0.75)
MMSE ( β = 0.50)
Sample
[Figure]
A. A DDITIONAL R ESULTS
More Qualitative Results.
In addition to the qualitative results presented in Fig. 2 in the main text, here we present more results for each considered dataset in Appendix Figs.
A.9-
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
A. R ESULTS ON N ATURAL I MAGES
We investigated the denoising performance of D IV N OISING network on the natural images benchmark dataset BSD 68 (Roth & Black, 2005) and show our results in Appendix Fig. 25, where the input has been corrupted with Gaussian noise of σ “ 25 .
With our depth 2 network having 96 feature channels in the ﬁrst network layer, we achieve a PSNR of 27 .
45 dB while our unsupervised N OISE 2V OID baseline gives 27 .
71 dB. As discussed in the main text, this does not come as a surprise since our D IV N OISING network is comparatively small and asked to learn a complete generative model of the entire data domain (see main text and Appendix Figs.
Learning such a model for the tremendous diversity present in natural images is challenging, and likely the reason why other architectures solving problems posed on the domain of natural images are much larger than our networks are.
Future versions of D IV N OISING will address this issue by using more expressive architectures.
However, D IV N OISING already gives us access to clean samples from the true (data) posterior (see Appendix Fig. 25).
[Figure]
that future work on D IV N OISING with larger networks and different network architectures/training schedules will expand the capabilities of this method to capture more image domains
A. H OW A CCURATE IS THE D IV N OISING M ODEL AND THE A PPROXIMATE P OSTERIOR ?
Upon close inspection, we ﬁnd that the images sampled by D IV N OISING exhibit various imperfections, making clear that they are in fact only samples from an approximate posterior.
For example, we ﬁnd that D IV N OISING samples are often smoother than real images, see e.g .
Appendix Figs.
14 and 21.
We attribute this problem to our network architecture (see also Appendix
Section A.9.
For instance, a U-N ET based supervised denoiser can make use of skip connections to propagate high frequency information.
But D IV N OISING VAEs have to pipe all information through the downsampled latent variable bottleneck.
Another common artefact in sampled images is the presence of faint overlayed structures in the background (see Suppl.
Fig. 23).
Note that this artefact is less pronounced than in the MMSE estimate (where we expect such artefacts).
We believe that most of these remaining issues will be solved/reduced by using more sophisticated network architectures and reﬁned training schedules.
A. D ERIVATION OF D IV N OISING L OSS F UNCTION FROM P ROBABILITY M ODEL P ERSPECTIVE
Here, we want to provide a more formal derivation of why our loss function can be used to train the
VAE as desired.
We follow a similar line of argument as has been laid out for the standard VAE by
Doersch in (Doersch, 2016).
In our framework, we assume that the observed data x is generated from some underlying latent variable z through some clean signal s via a known noise model p NM p
x
| s q .
This process of data generation is depicted as a graphical model shown in Appendix Fig.
[Equation]
The decoder describes a full joint model for all three variables:
[Equation]
In the assumed graphical model in (Appendix Fig. 26) x is conditionally independent of z given s .
Formally, this implies that
[Equation]
Using Supp.
Eq.
7, we can reformulate Supp.
Eq.
6 as
[Equation]
To train the generative model from Appendix Fig.
26 we try to adjust the parameters θ to maximize the likelihood of observing our training data x .
This means that we need to maximize
[Equation]
However, computing the integral in Supp.
Eq.
9 is intractable due to the high dimensionality of z .
In our particular model, we would need to integrate over 64 dimensions for each pixel for all our datasets except MNIST and KMNIST datasets where we would need to integrate over 8 dimensions for each pixel.
An alternative to computing the integral would be to approximate it by sampling a large number of values z 1 , z 2 , ..., z K from p p
z q and computing p θ p x q « 1 K ř
Kk
“ 1 p NM p x | s “ g θ p z k qq .
However, since p NM p x | s “ g θ p z k qq will be very close to 0 for almost all z k , this would require K to be a very large number for each image in our training set.
Following the idea introduced in (Kingma & Welling, 2014), we overcome this problem by instead using an encoder to describe an auxiliary distribution q φ p z | x q .
The encoder can take a noisy image x and yield a distribution over z values, which in turn are likely to produce x under the generative model.
We want the encoder distribution q φ p z | x q to approximate the true underlying distribution q φ p z | x q « p θ p z | x q , as it is implicitly described by our graphical model.
From Bayes theorem, p θ p z | x q factorizes as
[Equation]
The decoder in D IV N OISING setup is a deterministic function of z , i.e., g θ p z q “ s .
Hence, we can reformulate Supp.
Eq.
10 as
[Equation]
We can describe the quality of the encoder distribution, i.e .
how well it approximates the true p θ p z | x q via the KL divergence
[Equation]
Substituting Supp.
Eq.
11 in Supp.
Eq.
12, we get
[Equation]
Since ş q φ p z | x q d z “ 1 , we get
[Equation]
This implies
[Equation]
13) where ELBO is the Evidence Lower Bound as also introduced in (Kingma & Welling, 2019) in the context of standard VAEs and here ELBO “
[Equation]
d z .
Note that the KL divergence term in Supp.
Eq.
13 is always greater than or equal to 0 and hence, ELBO is a lower bound for log p θ p x q , i.e., log p θ p x q ě ELBO .
It follows from Supp.
Eq.
that
[Equation]
Supp.
Eq.
14 implies that maximizing ELBO with respect to φ and θ maximizes log p θ p x q and minimizes KL p q φ p z | x q|| p θ p z | x qq , the goals we seek to achieve.
Hence,
[Equation]
˘
[Equation]
Maximizing the ELBO is equivalent to minimizing the negative ELBO , thus giving us the
D IV N OISING loss function
[Equation]
where the expected value is approximated in each iteration by drawing a single sample from q φ p z | x q .
Note that the ﬁrst term in the summation in Supp.
Eq.
15 is the same as described in Section 5 in the main text whereas the second term in the summation is the same as used in the standard VAE loss.
A. C OMPARISON OF PREDICTED VARIANCES BY VARIOUS METHODS
Unsupervised D IV N OISING and vanilla VAEs are both trained fully unsupervised, learning to predict per pixel noise models.
Learning a good noise model is essential for good denoising performance as evident from Table 1.
Here, we compare the noise models and variance maps predicted for two datasets by our unsupervised D IV N OISING and vanilla VAEs.
BioID Face dataset.
This dataset has been synthetically corrupted with Gaussian noise of µ “ 0 and σ “ 15 .
[Figure]
Convallaria dataset.
This dataset is intrinsically noisy and the noise distribution resembles the shot noise and read out noise characteristics as typical for images acquired under low light settings.
[Figure]
as described in Prakash et al.
a much models learned/obtained by methods visually compare the denoising results show predicted varies the image
As a consequence of the approximately linear relationship between signal and noise variance both variance images closely resemble the denoised results.
However, additionally artifacts
A. Q UANTITATIVE COMPARISON OF D IV N OISING WITH S ELF 2S ELF
Since S ELF 2S ELF is trained per image, leading to prohibitive computation times on our test sets, we randomly chose single images for four of our datasets ( FU-PN2V Convallaria , FU-PN2V Mouse actin , FU-PN2V Mouse nuclei and W2S Ch.1 (avg1) ) which contain real-world noise.
We compare the performance of S ELF 2S ELF trained on single images with the performance of D IV N OISING when trained on p i q the same single image as S ELF 2S ELF , and p ii q the entire body of available noisy data in the respective dataset. All trained networks are then applied to the selected single images. Note that S ELF 2S ELF is run with its default settings.
Since S ELF 2S ELF training is computationally expensive even for a single image, we decided to limit training time to 10 hours per input on a NVIDIA TITAN
Xp GPU.
We monitored its performance by periodically computing the PSNR (every 3000 training steps), showing that even after 10 hours,
S ELF 2S ELF is not yet fully converged.
Table 2 shows all results we obtained.
It can be seen that
D IV N OISING , when trained on the full dataset, leads consistently to better performance, while
D IV N OISING trained on single images leads to comparable results in a fraction of training time and using signiﬁcantly less GPU memory.
[Table]
A. S AMPLING TIME DURING PREDICTION
During prediction, in order to obtain diverse results, or to compute the MMSE or MAP estimates
[Table]
we need to sample multiple denoised images from the trained D IV N OISING posterior.
Table 3 reports the time (in seconds) needed for sampling 1000 denoised images.
For all datasets holds that sampling 1000 denoised images requires less than 7 seconds.
[Table]