GAN2GAN: Generative Noise Learning for Blind Denoising with Single Noisy Images
Abstract
We tackle a challenging blind image denoising problem, in which only single distinct noisy images are available for training a denoiser, and no information about noise is known, except for it being zero-mean, additive, and independent of the clean image.
In such a setting, which often occurs in practice, it is not possible to train a denoiser with the standard discriminative training or with the recently developed Noise2Noise (N2N) training; the former requires the underlying clean image for the given noisy image, and the latter requires two independently realized noisy image pair for a clean image.
To that end, we propose GAN2GAN (Generated-Artiﬁcial-Noise to Generated-Artiﬁcial-Noise) method that ﬁrst learns a generative model that can 1) simulate the noise in the given noisy images and 2) generate a rough, noisy estimates of the clean images, then 3) iteratively trains a denoiser with subsequently synthesized noisy image pairs (as in N2N), obtained from the generative model.
In results, we show the denoiser trained with our GAN2GAN achieves an impressive denoising performance on both synthetic and real-world datasets for the blind denoising setting.
1. Introduction
Image denoising is one of the oldest problems in image processing and low-level computer vision, yet it still attracts lots of attention due to the fundamental nature of the problem.
A vast number of algorithms have been proposed over the past several decades, and recently, the CNN-based methods, e.g. , (Cha & Moon, 2019)
(Zhang et al., 2017)
(Tai et al., 2017) (Liu et al., 2018), became the throne-holders in terms of the PSNR performance.
The main approach of the most CNN-based denoisers is to apply the discriminative learning framework with (clean, noisy) image pairs and known noise distribution assumption.
While being effective, such framework also possesses a couple of limitations that become critical in practice; the assumed noise distribution may be mismatched to the actual noise in the data or obtaining the noise-free clean target images is not always possible or very expensive, e.g. , medical imaging (CT or MRI) or astrophotographs.
The more additional related works are explained in Supplementary Material (S.M.)
In this paper, we consider the pure unsupervised blind denoising setting where only single distinct noisy images are available for training.
Namely, nothing is known about the noise other than it being zero-mean, additive, and independent of the clean image, and neither the clean target images for blind training nor the noisy image pairs for N2N training is available.
The crux of our method is to ﬁrst learn a Wasserstein GAN (Arjovsky et al.
, 2017)-based generative model that can 1) learn and simulate the noise in the given noisy images and 2) generate rough, initially denoised images.
Using such generative model, we then synthesize noisy image pairs by corrupting each of the initially denoised images with the simulated noise twice and use them to train a CNN denoiser as in the N2N training ( i.e. , Noisy N2N).
We further show that iterative N2N training with reﬁned denoised images can signiﬁcantly improve the ﬁnal denoising performance.
We dubbed our method as GAN2GAN (Generated-Artiﬁcal-Noise to Generated-Artiﬁcial-Noise) and show that the denoiser trained with our method can achieve (sometimes, even outperform) the performance of the standard supervised-trained or N2N-trained blind denoisers for the white Gaussian noise case.
Furthermore, for mixture/correlated noise or real-world noise in microscopy/CT images, for which the exact distributions are hard to know a priori , we show our denoiser signiﬁcantly outperforms those standard blind denoisers.
2. Motivation
In order to develop the core intuition for motivating our method, we ﬁrst consider a simple, single-letter Gaussian noise setting.
Let Z =
X
+ N be the noisy observation of X ∼ N (0 , σ 2 X ) , corrupted by the N ∼ N (0 , σ 2 N ) .
It is well known that the minimum MSE (MMSE) estimator of
X given Z is f ∗ MMSE ( Z ) = E ( X |
[Equation]
) =
[Equation]
2 X
[Equation]
2 X + [Equation] 2 N Z. We now identify the optimality of N2N in this setting.
N2N Assume that we have two i.i.d.
copies of the noise N :
N 1 and N 2 .
Then, let Z
= X
+ N
and Z 2 = X
+ N 2 be the two independent noisy observation pairs of X .
The N2N in this setting corresponds to obtaining the MMSE estimator of Z 2 given Z 1 ,
[Equation]
in which (a) follows from N 2 being independent of Z 1 .
Note (1) has the exact same form as f ∗ MMSE ( Z ) , hence, estimating
X with f N2N ( Z ) also achieves the MMSE, in line with (Lehtinen et al., 2018).
“Noisy” N2N Now, consider the case in which we again have the two i.i.d.
N 1 and N 2 , but the noisy observations are of a noisy version of X .
Namely, let X (cid:48)
=
X + N 0 , in which N 0 ∼ N (0 , σ 20 ) , and denote Z (cid:48)
= X (cid:48)
+ N
and
Z (cid:48)
= X (cid:48)
+ N 2 as the noisy observation pairs.
Then, we can deﬁne a “Noisy” N2N estimator as the MMSE estimator of
Z (cid:48) 2 given Z (cid:48) 1 , f Noisy N2N ( Z (cid:48)
[Equation]
in which we denote y (cid:44) σ 20 /σ 2 X and assume that 0 ≤ y < 1 .
Note clearly (2) coincides with (1)
when
y = σ 20 = 0 .
Following N2N, (2) is essentially estimating X (cid:48) based on
Z (cid:48) =
X (cid:48)
+ N .
An interesting subtle question is what happens when we use the mapping f Noisy N2N ( Z, y ) for estimating X given Z = X + N , not X (cid:48) given Z (cid:48) .
Our theorem below, of which proof is in the Supplementary Material (S.M.), shows that for a sufﬁciently large σ 2 0 , f Noisy N2N ( Z, y ) gives a better estimate of X than X (cid:48) .
Theorem 1 Consider the single-letter Gaussian setting and f Noisy N2N ( Z, y ) obtained in (2).
Also, assume 0 < y < 1 .
Then, there exists some y 0 s.t.
∀ y ∈
( y 0 , 1) , E ( X − f Noisy N2N ( Z, y ))
2 < σ 20 .
Theorem 1 provides a simple, but useful, intuition that motivates our method; if simulating the noise in the images is possible, we may carry out the N2N training iteratively, provided that a rough noisy estimate of the clean image is initially available.
Namely, we can ﬁrst simulate the noise to generate noisy observation pairs of the initial noisy estimate, then do the Noisy N2N training with them to obtain a denoiser that may result in a better estimate of the clean image when applied to the actual noisy image subject to denoising (as in Theorem 1).
Then, we can reﬁne the estimates by iterating the Noisy N2N training with the generated noisy observation pairs of the previous step’s estimate of the clean image, until convergence.
To check whether above intuition is valid, we carry out a feasibility experiment and the experimental result is proposed in S.M.
3. Main Method: Three Components of GAN2GAN
To concretely describe our method, we ﬁrst set the notations.
We assume the noisy image Z is generated by Z = x + N , in which x denotes the underlying clean image and N denotes the zero-mean, additive noise that is independent of x .
For training a denoiser, we do not assume either the distribution or the covariance of N is known.
Moreover, we assume only a database of n distinct noisy images, D =
{ Z ( i ) } ni =1 , is available for learning a denoiser.
A CNN-based denoiser is denoted as ˆ X φ ( Z ) with φ being the model parameter, and we use the standard quality metrics, PSNR/SSIM, for evaluation.
Our method consists of three parts; 1) smooth noisy patch extraction, 2) training a generative model, and 3) iterative GAN2GAN training of ˆ X φ ( Z ) , each of which we elaborate below.
3.1. Smooth noisy patch extraction
The ﬁrst step is to extract the noisy image patches from
D that correspond to smooth, homogeneous areas.
Our extraction method is similar to that of the GCBD proposed in (Chen et al., 2018), but we make a critical improvement.
The GCBD determines a patch p (of pre-determined size) is smooth if it satisﬁes the following for all of its smaller sub-patches, q j , with some hyperparameters µ, γ ∈ (0 , 1) : [Equation]
[Equation]
in which E ( · ) and V ( · ) are the empirical mean and variance of the pixel values.
While (3) works for extracting smooth patches to some extent, it does not rule out choosing patches with highfrequency repeating patterns, which are far from being smooth.
Thus, we instead use the 2D discrete wavelet transform (DWT) for a new extraction rule; namely, we determine p is smooth if its four sub-band decompositions obtained by DWT, { W k ( p ) } 4 k =1 , satisfy
[Equation]
in which ˆ σ ( · ) is the empirical standard deviation of the wavelet coefﬁcients, E [ˆ σ W ( p )] (cid:44) 14 (cid:80) 4 k
=1 ˆ σ ( W k ( p )) , and λ ∈ (0 , 1) is a hyperparameter.
This rule is much simpler than (3), which has to be evaluated for all the sub-patches, { q j } .
Once N patches are extracted from D using (4), we subtract each patch with its mean pixel value, and obtain a set of “noise” patches, N = { n ( j ) } Nj =1 .
Such subtraction is valid since all the pixel values should be close to their mean in a smooth patch, and the noise is assumed to be zero-mean, additive.
3.2. Training a W-GAN based generative model
Equipped with D =
{ Z ( i ) } ni =1 and the extracted noise patches N = { n ( j ) } Nj =1 , we train a generative model, which can learn and simulate the noise as well as generate initial noisy estimates of the clean images, hence, realize the Noisy N2N training explained in Section 2.
As shown in Figure 2 (in S.M.), our model has three generators, { g θ 1 , g θ 2 , g θ 3 } , and two critics, { f w 1 , f w 2 } , in which the subscripts stand for the model parameters.
The loss functions associated with the components of our model are:
[Equation]
The loss (5) is a standard W-GAN (Arjovsky et al., 2017) loss for training the ﬁrst generator-critic pair, ( g θ 1 , f w 1 ) , of which g θ 1 learns to generate the independent realization of the noise mimicking the patches in N = { n ( j ) } Nj =1 , taking the random vector r ∼ N (0 , I ) as input.
The second loss (6) links the two generators, g θ 1 and g θ 2 , with the second critic, f w 2 .
The second generator g θ 2 is intended to generate the estimate of the underlying clean patch for
Z , i.e. , coarsely denoise Z , and the critic f w 2 determines how close the distribution of the generated noisy image, g θ 2 ( Z ) + g θ 1 ( r ) , is to the that of Z 1 .
Our intuition is, if g θcan realistically simulate the noise, then enforcing g θ 2 ( Z ) + g θ 1 ( r ) to mimick Z would result in learning a reasonable initial denoiser g θ 2 .
One important detail regarding g θ 2 is its ﬁnal activation must be the sigmoid function for stable training.
The third loss (7), which resembles the cycle loss in (Zhu et al., 2017), imposes the encoder-decoder structure between g θ 2 and g θ 3 , hence, helps g θ 2 to compress the most redundant part of Z , i.e. , the noise, and carry out the initial denoising.
Once the losses are deﬁned, training the generators and critics are done in an alternating manner, as in the training of W-GAN (Arjovsky et al., 2017), to approximately solve
[Equation]
in which ( α, β, γ ) are hyperparameters to control the tradeoffs between the loss functions.
There are a couple of subtle points for training with the overall objective (8), and we describe the full details on model architectures, hyperparameters, and training procedures in the S.M.
3.3. Iterative GAN2GAN training of a denoiser
With our generative model, we then carry out the iterative
Noisy N2N training as described in Section 2, with the generated noisy images.
Namely, given each Z ( i ) ∈ D , we generate the pair
[Equation]
in which r ( i ) 11 , r ( i ) 12 ∈ R 128 are i.i.d.
∼ N ( 0 , I ) .
In contrast to the ideal case in Section 2, each generated image in (9) is a noise-corrupted version of g θ 2 ( Z ( i ) ) , in which the corruption is done by the simulated noise g θ 1 ( r ) .
Denoting the set of such pairs as ˆ D 1 = { (ˆ Z ( i ) 11 , ˆ Z ( i ) 12 ) } ni =1 , a denoiser ˆ X φ ( Z ) is trained by minimizing L G2G ( φ , ˆ D 1 ) (cid:44) 1 n (cid:80) ni
=
( ˆ Z ( i ) 11 − ˆ X φ
( ˆ Z ( i ) 12 )) 2 .
In L G2G ( · ) , we only use the generated noisy images and do not use the actual observed
Z ( i ) , hence, we dubbed our training as GAN2GAN (G2G) training.
Now, denoting the learned denoiser as G2G 1 (with parameter φ 1 ), we can iterate the G2G training.
For the j -th iteration (with j ≥ 2 ), we generate
[Equation]
for each Z ( i ) and denote the resulting set of the pairs as ˆ D j .
Note in (10), we update the noisy estimate of the clean image with the output of G2G j − 1 .
Then, the new denoiser G2G j is obtained by computing φ j (cid:44) arg min φ L G2G ( φ , ˆ D j ) , where the minimization is done via warm-starting from φ j − 1 .
4. Experimental results 4.1. Data and experimental settings
Data & training details In synthetic noise experiments, we always used the noisy training images from BSD (BSD) (Martin et al., 2001).
For evaluation, we used the standard BSD68 (Roth & Black, 2009) as a test set.
For real-noise experiment, we experimented on two data sets: the WF set in the microscopy image datasets in (Zhang et al., 2019) and the reconstructed CT dataset.
For both datasets, we trained/tested on each (Avg = n ) and each dose level, respectively, which corresponds to different noise levels.
The details on the experimental settings and baslines is proposed in S.M.
4.2. Denoising results on synthetic noise
White Gaussian noise Table 1 shows the results on BSD corrupted by white Gaussian noise with different σ ’s. Several variations of our G2G, g θ 2 and the G2G iterates,
G2G j ≥ 1 , are shown for two different training data versions [Table] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
for learning the generative model
[Figure] Firstly [Table] we clearly observe the iterative G2G training is very effective; namely, it signiﬁcantly improves the initial noisy estimate g θ 2 , particularly when the quality of the initial estimate is not good enough.
Secondly, we note G2G 1 already considerably outperforms N2V (Krull et al., 2019), which is trained with the exact same model architecture and dataset.
Finally, the performance of G2G 3 is very strong; it outperforms BM3D (Dabov et al., 2007), which knows true σ , and even sometimes outperforms the blindly trained DnCNN-B (Zhang et al., 2017) and N2N (Lehtinen et al., 2018), which is trained with the same BSD400 dataset, but with more information.
This somewhat counter-intuitive result is possible since our G2G j accurately learns the correct noise level in the image, while DnCNN-B and N2N are trained with the composite noise levels, σ ∈ [0 , 55] .
Mixture and correlated noise Table 2 shows the results on mixture and correlated noise beyond white Gaussian.
Note our G2G j does not assume any distributional or correlation structure of the noise, hence, it can still run as long as the assumption on the noise holds.
In the table, the G2G results are for (BSD) as speciﬁed above.
Moreover, DnCNN-B and N2N are still blindly trained with the mismatched white
Gaussian noise.
From the table, we ﬁrst note that DnCNN-B and N2N suffer from serious performance degradation for both mixture and correlated noises due to noise mismatch, and the conventional BM3D outperforms them for some cases ( e.g. , Case A for mixture noise).
However, we note our G2G 2 can still denoise very well after just two iterations and outperforms all the baselines for all noise types, solely based on single noisy images.
Note N2V seriously suffers and is not comparable to ours.
Finally, N2C(Eq.(4)) is a sound upper bound for all noise types, conﬁrming the correctness of the extraction rule (4).
4.3. Denoising results on real noise
We also test our method on the real-world noise.
While some popular real noise is known to have source-dependent characteristics, there are also cases in which the noise is sourceindependent and pixel-wise correlated, which satisﬁes the assumption of our method.
We tested on two such datasets, the Wide-Focal (WF) set in the microscopy image dataset (Zhang et al., 2019) and a Reconstructed CT dataset.
A more detailed description and analysis on these two datasets are in S.M.
The WF and Reconstructed CT data has 5 sets (Avg
[Figure]
= 1 , 2 , 4 , 8 , 16 ) and 4 sets (Dose= 25 , 50 , 75 , 100 ) with different noise levels, respectively.
We did not exploit the fact that the images are multiple noisy measurements of a clean image, which enables employing N2N, but treated them as noisy images of distinct clean images.
Figure 1(a) and 1(b) shows the PSNR of all methods for each dataset, respectively, averaged over all sets.
The baselines were DnCNN-B and BM3D and N2V.
We note BM3D estimated noise σ using the method in (Chen et al., 2015).
We iterated until
G2G 3 and N2C(Eq.(4)) was an upper bound for each set.
We clearly observe that the performance of G2G j signiﬁcantly improves (over g θ 2 ) as the iteration continues.
In results,
G2G 3 becomes signiﬁcantly better than DnCNN-B and N2V as well as BM3D, still one of the strongest baselines for real-world noise denoising when no clean target images are available, for both datasets.
We report more detailed experimental results (including SSIM) on both datasets in S.M.
Moreover, the inference time for BM3D is about 4.5 ∼ 5.
seconds per image since a noise estimation has to be done for each image separately, whereas that for G2G j is only 4 ms (on GPU), which is another signiﬁcant advantage of our method.
The visualizations on the denoising result are shown in the S.M.
1. Related work
Several works have been proposed to overcome the limitation of the vanilla supervised learning based denoising.
As mentioned above, Noise2Self (N2S) (Batson & Royer, 2019) and Noise2Void (N2V) (Krull et al., 2019) recently applied self-supervised learning (SSL) approach to train a denoiser only with single noisy images.
Their settings exactly coincide with ours, but we show later that our GAN2GAN signiﬁcantly outperforms them.
More recently, (Laine et al., 2019) improved N2V by incorporating speciﬁc noise likelihood models with Bayesian framework, however, their method required to know the exact noise model and could not be applied to more general, unknown noise settings.
Similarly, (Soltanayev &
Chun, 2018) proposed SURE (Stein’s Unbiased Risk Estimator)-based denoiser that can also be trained with single noisy images, but it worked only with the Gaussian noise.
Their work was extended in (Zhussip et al., 2019), but it required noisy image pairs as in N2N as well as the Gaussian noise constraint.
[Table]
al., 2019) (cid:55) (cid:55) (cid:55) GAN2GAN (Ours) (cid:55) (cid:55) (cid:55) (Chen et al., 2018) devised GCBD method to learn and generate noise in the given noisy images using W-GAN (Arjovsky et al., 2017) and utilized the unpaired clean images to build a supervised training set.
Our GAN2GAN is related to (Chen et al., 2018), but we signiﬁcantly improve their noise learning step and do not use the clean data at all.
Table 1 summarizes and compares the settings among the above mentioned recent baselines.
We clearly see that only our GAN2GAN and N2V do not utilize any “sidekicks” that other methods take advantage of.
More classical denoising methods typically are capable of denoising solely based on the single noisy images by applying various principles, e.g. , ﬁltering-based (Buades et al., 2005; Dabov et al., 2007), optimization-based (Elad & Aharon, 2006;
Mairal et al., 2009), Wavelet-based (Donoho & Johnstone, 1995), and effective prior-based (Zoran & Weiss, 2011).
Those methods typically are, however, computationally intensive during the inference time and cannot be trained from a separate set of noisy images, which limits their denoising performance.
Another line of recent work worth mentioning is the deep learning-based priors or regularizers, e.g. , (Ulyanov et al., 2018;
Yeh et al., 2018; Lunz et al., 2018), but their PSNRs still fell short of the supervised trained CNN-based denoisers.
2. Proof of Theorem
Theorem 1 Consider the single-letter Gaussian setting and f Noisy N2N ( Z, y ) obtained in (Eq.(2), manuscript).
Also, assume 0 <
y
=
σ 20 /σ 2 X < 1 .
Then, there exists some y 0 s.t.
∀ y ∈
( y 0 , 1) , E ( X − f Noisy N2N ( Z, y ))
2 < σ 20 .
Proof: We consider the following chain of equalities:
[Equation]
Now
[Equation]
by denoting the numerator of (5) as g ( y ) [Equation] we have
[Equation]
Then, we can easily see that
[Equation]
Therefore, in 0 < y < 1 , we can see g ( y ) is an increasing function and has a root y 0 in the interval.
Hence, the claim of the theorem: for all y ∈ ( y 0 , 1) , E ( X − f Noisy N2N ( Z, y )) 2 < σ 20 holds.
3. The feasibility experiment on NoisyN2N
[Figure]
Figure 1 shows the denoising results on BSD68 (Roth & Black, 2009) for Gaussian noise with σ = 25 .
The blue line is the
PSNR of the N2N model trained with noisy observation pairs of the clean images in the BSD training set, serving as an upper bound.
The orange line, in contrast, is the PSNR of the Noisy N2N 1 model that is trained with the noisy observation pairs of the noisy estimates for the clean images, which were set to be another Gaussian noise-corrupted training images.
The standard deviations ( σ 0 ) of the Gaussian for generating the noisy estimates are given in the horizontal axis, and the corresponding PSNRs of the estimates are given in the parentheses.
Although Noisy N2N 1 clearly lies much lower than the
N2N upper bound, we note its PSNR is still higher than that of the initial noisy estimates, which is in line with Theorem 1.
Now, if we iterate the Noisy N2N with the previous step’s denoised images ( i.e. , Noisy-N2N 2 /Noisy-N2N 3 for second/third iterations, respectively), we observe that the PSNR signiﬁcantly improves and approaches the ordinary N2N for most of the initial σ 0 values.
Thus, we observe the intuition from Theorem 1 generalizes well to the image denoising case in an ideal setting, where the noise can be perfectly simulated, and the initial noisy estimates are Gaussian corrupted versions.
The remaining question is whether we can also obtain similar results for the blind image denoising setting.
We show our generative model-based approach in details in the next section.
4. Overall structure of the W-GAN based generative model
[Figure]
5. Details on the experimental settings 5.1. Training details
For the generative model training, the patch size used for D and N was 96 × 96 , and n and N were set to 20 , 000 (BSD) and 40 , 000 (microscopy), respectively.
For the iterative G2G training, the patch size for D was 120 × 120 and n = 20 , 500 , and in every mini-batch, we generated new noisy pairs with g θ 1 as in the noise augmentation of (Zhang et al., 2017).
The architecture of G2G j was set to 17-layer DnCNN in (Zhang et al., 2017).
5.2. Baselines
The baselines were BM3D (Dabov et al., 2007), DnCNN-B (Zhang et al., 2018), N2N (Lehtinen et al., 2018), and N2V (Krull et al., 2019).
We reproduced and trained DnCNN-B, N2N and N2V using the publicly available source codes on the exactly same training data as our iterative G2G training.
For DnCNN-B and N2N, which use either clean targets or two independent noisy image copies, we used 20-layers DnCNN model with composite additive white Gaussian noise with σ ∈ [0 , 55] .
N2V considers the same setting as ours and uses the exact same architecture as G2G j ; more details on N2V are also given in the S.M. We could not compare with the scheme in (Laine et al., 2019), since their code cannot run beyond white Gaussian noise case in our experiments and they had an unfair advantage: they newly generate noisy images by corrupting given clean images for every mini-batch whereas we assume the given noisy images are ﬁxed once for all.
It is known that such noise augmentation signiﬁcantly can increase the performance, and their code could not run in our setting in which the noisy images are ﬁxed once given.
As an upper bound, we implemented N2C(Eq.(4)), denoting a 17-layer
DnCNN trained with clean target images in BSD400 and their noisy counterpart, which is corrupted by our g θ 1 learned with Eq.(4) in the manuscript.
5.3. Details on Mixture noise
For mixture noise, we tested with two cases.
Case A corresponds to the same setting as given in (Chen et al., 2018), i.e. , 70% ∼ N (0 , 0 .
1 2 ) , 20% ∼ N (0 , 1) , and 10% ∼ Unif [ − s, s ] with s = 15 , 25 .
For case B, we tested with larger variances, i.e. , 70% Gaussian N (0 , 15 2 ) , 20% Gaussian N (0 , 25 2 ) , and 10% Uniform [ − s, s ] with s = 30 , 50 .
5.4. Details on Correlated noise
For correlated noise, we generated a spatially correlated Gaussian noise, of which neighboring k × k blocks are correlated and marginal standard deviation σ = 15 , 25 .
For this, we generated the following noise for each (cid:96) -th pixel,
[Equation]
in which { M (cid:96) } are white Gaussian N (0 , σ 2 ) , NB (cid:96) is the k × k neighborhood patch except for the pixel (cid:96) , and η is a mixture parameter.
We set η = 1 / √ 2 such that the marginal distribution of N (cid:96) is also N (0 , σ 2 ) and set k = 16 .
Note in this case, N (cid:96) has a spatial correlation, and we tested with σ = 15 , 25 .
5.5. Smooth noisy patch extraction 5.5.1. GCBD (C HEN ET AL ., 2018) RULE E Q .(3)
The original GCBD paper (Chen et al., 2018) did not provide any source code or training data, hence, we reproduced their noisy patch extraction algorithm.
There are six hyperparameters for the rule [Eq.(3), Manuscript], and we used the exact same hyperparameters given in their paper, which are shown in Table 2.
d and h denote the size of a patch, p , and its sub-patches, q j , given in [Eq.(3), Manuscript], respectively.
s p and s q are the stride sizes for extracting the patches, p and { q j } , from a given image.
µ and λ are the hyperparameters of the rule for selecting the smooth patches shown in [Eq.(3),
Manuscript].
[Table]
γ
Values    0.
[Table]
5.5.2. G2G RULE E Q .(4)
There are three hyperparemeters for our extraction rule [Eq.(4 ), Manuscript], λ , d (the patch size), and s d (the stride size for extracting patches from an image).
The choices for our experiments are shown in Table 4.
Moreover, we stress that we did not tune λ using clean images, but the different λ values in the table are determined by the pre-determined number of extracted patches by applying our rule [Eq.(4), Manuscript].
Moreover, as argued in Section 3.1 (manuscript), we do not require any sub-patches to be extracted, hence, have only half the hyperparameters compared to the GCBD rule.
[Table]
5.5.3. E FFECT OF λ
Table 4 shows the effect of λ in [Eq.(4), Manuscript] on the ﬁnal performance of G2G 2 .
Note the smaller the λ , the less number of patches are extracted, but the homogeneity increases.
The table shows λ clearly affects the denoising performance of g θ 2 , but as the iterative G2G training continues, the performance of G2G Hence, in our experiments [Table]
we did not optimize λ based on any clean validation set, but just set λ based on the number of extracted patches and checking the visual qualities of the patches.
[Table]
5.6. Training a W-GAN based generative model
Here [Table] we elaborate a couple of subtle points for training our generative model as mentioned in Section 3.2 (manuscript) [Table]
Firstly, given the overall optimization objective [Eq.(8), manuscript] [Table] we use ( α, β, γ ) =
(1 , 1 , 0) for the inner maximization for critics, and use ( α, β, γ ) = (5 , 1 , 10) for the outer minimization for generators.
The main intuition for using different ( α, β, γ ) for training the generators is due to different levels of conﬁdence in the generator loss terms.
Namely, we assign the largest weight to [Eq.(7), Manuscript] since it is a deterministic loss and its value has a clear meaning.
The generator loss [Eq.(5), Manuscript], which is in the form of the standard W-GAN loss, gets the medium level weight since the meaning of its value is less certain than [Eq.(6), Manuscript].
In contrast, the generator loss in [Eq.(5), Manuscript], which consists of two generators, can become somewhat unstable during training, hence, it gets the least weight.
Figure 10(a) compares the performance of g θ 2 ’s on BSD68 ( σ = 25) when using ( α, β, γ ) =
(5 , 1 , 10) , for the outer minimization, as proposed, and using ( α, β, γ ) =
We observe there is a signiﬁcant gap between the two.
[Figure]
Secondly, the output layer of g θ 2 must have the sigmoid activation function.
Note g θ 2 itself can be thought of another denoiser, but since we are not training it with any target, we need to ensure the outputs of g θ 2 have values between [0 , 1] to prevent from obvious errors of generating negative or out-of-bound pixel values.
Without the sigmoid activation, it turned out all the generators cannot be trained properly at all.
Finally, using the right architectures for the generators and critics, e.g. , number of layers and ﬁlters, was critical since the training procedure got very sensitive to the architectural variations.
Tables 5 shows the details on the architecture of our ﬁrst generator, g θ 1 , which aims to generate noise patches.
The dimension of r (the input random vector) was set to 128, and
C denotes the channel of the generated noise patch.
The architectures of the g θ 2 and g θ 3 in our generative model are equal to that of the DnCNN model (Zhang et al., 2018), however, g θ 2 had 15 layers with sigmoid activation in the output layer, and g θ 3 had 17 layers and linear activation in the output layer.
In addition, the architectures of the two critics, { f w 1 , f w 2 } , in our generative model are given in Table 6.
For training, we carry out the random cropping of the given patches to the size of 64 × 64 , and the data augmentation was done by ﬂipping the cropped patches horizontally and vertically.
For optimization, we used Adam (Kingma & Ba, 2015) optimizer for the three generators and RMSProp (Tieleman & Hinton, 2012) optimizer for the two critics.
The initial learning rates were set to 0 .
0004 and 0 .
0005 for Adam and RMSProp, respectively.
Also, the learning rate decay, dropping
[Table]
[Table]
[Table]
the learning rate linearly starting from epoch 10, is applied to the Adam optimizer.
The parameter clipping was done for the critics and the range was set to [ − 0 .
02] , and the number of training iterations for the critics was 5.
The total number of training epochs was 30 and the mini-batch size was 64.
The pseudo algorithm for training a generative model is in Algorithm
Algorithm 1 Training a generative model, all experiments in this paper used the defaults values, n critic = 5 , n epoch = 30 , m = 64, α g = 4 e − 4 , α critic =
5 e − 5 , α = 5 , β = 1 , γ =1: Require D , λ 2: N ← NoisePatchExtraction ( D , λ ) 3: for ep GAN ← 1 , n epoch do 4:
Sample { n ( i ) } mi =1 ∼ N , { r ( i ) } mi =1 ∼ N (0 , I ) , { Z ( i ) } mi =1 ∼ D 5: for ep critic ← 1 , n critic do 6: g w 1 ← ∇ w
[ L n ( θ 1 , w 1 )] 7: g w 2 ← ∇ w
[ L Z ( θ 1 , θ 2 , w 2 )] 8: w 1 ← Clip ( w 1 + α critic · Adam ( w 1 , g w 1 ) , − c, c ) 9: w 2 ← Clip ( w 2 + α critic · Adam ( w 2 , g w 2 ) , − c, c ) 10: end for 11: g θ 1 , g θ 2 , g θ 3 ← ∇ θ 1 ,θ 2 ,θ
[ α L n ( θ 1 , w 1 ) + β L Z ( θ 1 , θ 2 , w 2 )
+ γ L cyc ( θ 2 , θ 3 ) ] 12: θ 1 ← θ 1 − α g · Adam ( θ 1 , g θ 1 ) 13: θ 2 ← θ 2 − α g · Adam ( θ 2 , g θ 2 ) 14: θ 3 ← θ 3 − α g · Adam ( θ 3 , g θ 3 ) 15: end for 16: return g θ 1 , g θ
5.6.1. A BLATION STUDY ON L CYC
As shown in the synthetic noise case of Figure 6(b) (manuscript), the iterative G2G training is powerful such that there is a negligible performance difference between the schemes with and without g θ 2 , when the number of iterations is sufﬁciently large.
Consequently, the cycle loss L cyc also does not have signiﬁcant effect in the ﬁnal performance for the synthetic noise case.
However, for the real noise case, L cyc becomes more critical.
As shown in Figure 4(a) and 4(b), on WF(Avg = 1 ) dataset, we observe that when there is no L cyc in our generative model, the ﬁnal PSNR or SSIM performances cannot reach the model with L cyc even after many iterations of G2G training.
Hence, this result shows the necessity of L cyc .
[Figure]
training of a denoiser
We do the same random cropping and data augmentation as in the generative model training.
Moreover, for every minibatch in the G2G training, we generated new synthetic noisy image pairs using our trained generators as was done in the noise augmentation of (Zhang et al., 2017).
Adam optimizer with an initial learning rate 0 .
001 was used, and the learning rate scheduling, which halves the learning rate every 20 epochs, was applied.
The total number of training epochs was 50, and the mini-batch size was 4.
We also stress that we set the architecture of ˆ X θ ( Z ) identical to that of 17-layers DnCNN in (Zhang et al., 2017) to make a fair comparison.
The pseudo algorithm for training a generative model is in Algorithm
Algorithm 2 Training G2G, all experiments in this paper used the defaults values, n epoch = 50 , α G 2 G = 1 e −1: Require D , g θ 1 , g θ 2 , φ , num iter, m 2: for j ← 1 , num iter do 3: for ep ← 1 , n epoch
Noise2Void
We used the publicly available source code of Noise2Void (N2V)
(Krull et al., 2019) to obtain the denoising results of N2V.
Most of the hyperparameters were set to the default ones, but we changed three things to make a fair comparison with our method.
Firstly, while the CNN architecture for the original N2V was a UNet3, we used the DnCNN (Zhang et al., 2018) withlayers such that it has the same structure as our G2G model.
Secondly, as also is done in (Krull et al., 2019), we had to use a validation set to do a proper model selection for N2V (i.e., the best epoch), while our G2G does not require any validation set (since we always use a model at the last epoch).
The reason why N2V needs a validation is that its learning curve is very unstable and a proper model selection greatly affects the ﬁnal denoising performance.
To that end, since we used 20,500 patches with 120 × 120 size for training our G2G and other baselines, we divided the 20,500 patches into 18,000 training patches and 2,500 validation patches for training and selecting the best N2V model.
Thirdly, we set ’mini batch size’ to 4 (as our G2G) and ’train steps per epoch’ to ’num of training data / mini batch size’, hence, 4,500.
Other hyperparameters are given in Table 7.
[Table]
6. Comparison of the patch extraction rules
Here, we make a further, thorough comparison between the GCBD smooth patch extraction rule [Eq.(3), manuscript] and ours [Eq.(4), manuscript].
We selected three noisy patches from [Figure 2(b), manuscript] and show the decision criterion of each rule for each image Figure 5.
From the ﬁgure, we can observe that while our G2G rule correctly excludes the patches in Figure 5(b) and 5(c) as non-homogeneous patches, the GCBD rule wrongly determines them also as homogeneous patches.
That is, we note that since the DWT transform used in our rule can successfully disaggregate the high and low frequency components in the patches, the patches with self-similar repeating patterns would have signiﬁcantly varying sub-band coefﬁcient variances as shown in the ﬁgures.
Hence, our rule can exclude those patches.
However, in the GCBD rule, there may exist a sub-patch q j that has similar empirical mean and variance as the original patch p , thus, it may determine the patches with the self-similar repeating patterns as homogeneous as well.
We believe these examples clearly show the stark difference between our rule and the GBCD rule for smooth patch extraction.
[Figure]
[Table]
[Figure]
[Figure]
[Figure]
[Figure]
7. Result table for real microscopy noise
We report the detailed experimental results on the real microscopy images in Table 8.
We observe that Iterative G2G increases PSNR/SSIM in WF.
[Table]
8. Description and the result table on Reconstructed CT dataset
The reconstructed CT dataset consists of chest and head parts of 27 pediatric extended cardiac-torso phantoms (Segars et al., 2015), which provide a highly realistic model of the human anatomy.
We extracted 60 image slices from each phantom, leading to 1620 image slices in total.
The dataset was generated in the following procedure.
First, noiseless projection data were acquired in a parallel-beam geometry with Siddon’s ray-driven algorithm (Sidky & Pan, 2008).
To reduce view aliasing artifacts, the detector quarter-offset was used during a CT scan.
Second, Poisson noise was generated and added to the noiseless projection data.
Note that the mean number of detected photons was set to 2,500, 5,000, 7,500, and 10,000 to simulate 25%, 50%, 75%, and 100% of a normal dose, respectively.
Finally, the images were reconstructed by ﬁltered backprojection (Hsieh, 2003).
To preserve ﬁne anatomical structures in the images, the Ram-Lak ﬁlter was used as a reconstruction ﬁlter.
Detailed simulation parameters are summarized in Table 9.
[Table]
We divided 27 phantoms into training and test data and the phantom number for each dataset is in Fig 10.
Also, We visualized the ﬁrst image of Female 1 in Fig 6.
We can clearly see that each dose has a different noise level, and the noise is source independent and correlated.
Finally, Table 11 shows the details of experimental results on Reconstructed CT dataset.
[Table]
[Table] [Figure]
9. Analysis on real microscopy image
In this section, we analyze why our G2G also works well on the real microscopy image dataset (WF)
(Zhang et al., 2019).
although the source-dependent noise does not satisfy our assumption on the noise.
The real microscopy image dataset consists of three different types of dataset, which are Wide-Focal(WF), Two-Photon(TP) and Con-Focal(CF), and It is generally known that the real noise follows the Poisson-Gaussian model (Zhang et al., 2019),
Z i = x i + N i , i = 1 , 2 , . . .
, (11) in which N i ∼ N (0 , σ 2 i ) and
[Equation]
(12) with a scaling factor α > 0 .
Thus, the noise variance depends on the underlying clean source pixel value, and α determines the level of the dependence.
Clean image
Corrupt image
Noise
W i de - F o c a l ( A v g = 1 )
In Table 8, we observe that our G2G performs well for WF compared to other baselines, hence, we visualize clean, noisy, noise images from each set and examine if there are any notable difference in the noise distributions.
Figure 7 shows two image samples (Avg = 1 cases) from the Wide-Focal (WF) set.
The noise images are obtained by subtracting the clean images from its noisy versions.
Note even though the intensities in the source images change signiﬁcantly among pixels (particularly for the top image), the noise images do not show any source-dependent patterns.
Hence, we can deduce that α may be small for the WF images.
Also, we could see that there is a correlated pattern in the noise.
We belive that these back the good performance of G2G for the WF set.
Figure 8, on the other hand, visualizes an image from TP set for Avg = { 1 , 16 } cases.
Comparing with Figure 7, we can clearly see the source-dependent patterns in the noise images, particularly severely for the Avg = 1 case.
Also, CP set showed the similar source-dependent noise patterns.
This source-dependent noise is not in our assumption so we did not apply GAN2GAN to TP and CP.
However, we want to stress out that the source independent real noise also exists and
GAN2GAN shows the best result compared to any other baselines.
Clean image Corrupt image Noise
T w o - P ho t on ( A v g = 1 )
T w o - P ho t on ( A v g = 16 )
10. Ablation study
[Figure]
Noise patch extraction Here, we evaluate the effect of the noisy patch extraction rules (3) and (4) (in the manuscript) in the ﬁnal denoising performance.
Figure 9 compares the PSNR of N2C(GCBD Eq.(3)), a re-implementation of (Chen et al., 2018), N2C(Ours Eq.(4)) and the best G2G, for each dataset.
We note neither source code nor training data of (Chen et al., 2018) is publicly available, and the PSNR in (Chen et al., 2018) could not be reproduced (with the exact same η and γ as in (Chen et al., 2018)).
From the ﬁgure, we clearly observe the signiﬁcant gap between N2C(Our Eq.(4)) and N2C(GCBD Eq.(3)), particularly when the noise is not white Gaussian.
Moreover, our pure unsupervised G2G with (4) even outperforms N2C(GCBD Eq.(3)) that utilizes the clean target images.
Figure 10(a) shows the PSNRs of g θ 2 on BSD68/Gaussian( σ = 25 ) trained with three variations; “No L Z ” for no f w 2 , “No L cyc ” for no (7) in the manuscript and g θ 3 , and “No sigmoid” for no sigmoid activation at the output layer of g θ 2 .
We conﬁrm that our proposed architecture achieves the highest PSNR for g θ 2 , the sigmoid activation and f w 2 are essential, and the cylce loss (7) in the manuscript is also important.
Achieving a decent PSNR for g θ 2 is beneﬁcial for saving the number of G2G iterations and achieving high ﬁnal PSNR.
More detailed analyses on the generative model architecture are in the S.M. Figure 10(b) and 10(c) show the effect of the quality of the
[Figure]
initial estimate for the iterative G2G training.
From Figure 1, one may ask whether g θ 2 is indeed necessary, since even when σ 0 ≈ σ , the iterating the Noisy N2N can mostly achieve the upper bound.
Hence, for samples of synthetic and real microscopy data, we evaluate how G2G j performs when the iteration simply starts with Z .
Figure 10(b) shows a somewhat surprising result that for synthetic noises, starting from Z achieves essentially the same performance as starting from g θwith a couple more G2G iterations.
However, for real microscopy noise case in Figure 10(c), WF(Avg = 1 ) in which starting from Z achieves far lower performance than starting from g θ 2 , justifying our generative model for attaining the initial noisy estimate.
11. Visualizations 11.1. Visualization of ˜ Z
Figure 11 and 12 visualize the simulated noisy image pairs (ˆ Z 1 , ˆ Z 2 ) , generated from our generative model, for synthetic and real noise cases, respectively.
A close examination shows that the images are not simple copies of the original noisy image Z but are successfully synthesized with the independent noise processes.
G au ss i an ( 𝜎 = ) 𝑍
M i x t u r e ( 𝑠 = 3)
C o rr e l a t ed ( 𝜎 = 2) [Figure] [Equation]
𝑍 𝑍
[Equation]
! 𝑍" "
W
F ( A v g = 4 )
Figure 13 visualizes the denoising results of a BSD68 image for different types of noise.
Note the clear difference in the noise characteristics for Gaussian, mixture, and correlated noises.
The visualization of G2G 3 certiainly seems better than
N2V and BM3D, in line with the PSNR results.
DnCNN-B and Noise2Noise use more information than G2G 3 , but the visualzation as well as the PSNR of G2G 3 are comparable to those of the two methods.
Noisy image
[Equation]
G a u ss i a n n o i s e 𝜎 =
M i x
t u r e n
o
i s e 𝑠 =
C o rr e l a t e d n
o
i s e 𝜎 =
PSNR : 22.
PSNR : 25.02 PSNR : 14.
PSNR : 23.
PSNR : 24.58 PSNR : 24.
PSNR : 29.
PSNR : 29.73 PSNR : 23.
PSNR : 23.
PSNR : 30.09 PSNR : 29.
PSNR : 26.
PSNR : 27.37 PSNR : 22.
PSNR : 25.
PSNR : 27.76 PSNR : 27.
11.3. Additional visualizations on the real microscopy images
We also visualize additional denoised images of WF images in Figure 14.
We can see that the denoising results of the baselines for WF (Avg = 1) are very noisy, but G2G 3 shows relatively clean denoising results than others. W
F ( A v g = 4 )
Clean
BM3D
N2V
G2G
W
F (
A v g = 1 ) 24.02 / 0.
29.37 / 0.6243 PSNR / SSIM 24.76 / 0.
32.14 / 0.8443 PSNR / SSIM 29.86 / 0.
W
[Equation]
[Equation] = [Equation]
35.14 / 0.8509 PSNR / SSIM
11.4. Additional visualizations on the Reconstructed CT
We visualize the denoising result of a Reconstructed CT image in Figure 15.
We observed that BM3D and N2V shows a still noisy result on this image but G2G 3 shows a clearly denoised result.
Clean
BM3D
N2V
G2G
C T ( D o s e = ) 32.64 / 0. 45.88/ 0.9394 PSNR [Figure]