NERDS: A G ENERAL F RAMEWORK TO T RAIN C AMERA D ENOISERS FROM R AW -RGB N OISY I MAGE P ARIS
Anonymous authors
Paper under double-blind review
A BSTRACT
We aim to train accurate denoising networks for smartphone/digital cameras from raw-RGB noisy image pairs.
Downscaling is commonly used as a practical denoiser for low-resolution images.
Based on this processing, we found that the pixel variance of natural images is more robust to downscaling than the pixel variance of camera noise.
Intuitively, downscaling removes high-frequency noise more easily than natural textures.
To utilize this property, we can adopt noisy/clean image synthesis at low-resolution to train camera denoisers.
On this basis, we propose a new solution pipeline – NERDS that estimates camera noise and synthesizes noisy-clean image pairs from only noisy images.
In particular, it first models the noise in raw-sensor images as Poisson-Gaussian distributions, then estimates noise parameters using the difference of pixel variances by downscaling.
We formulate the noise estimation as a gradient-descent-based optimization problem through a reparametrization trick.
We further introduce a new Image Signal Processor (ISP) estimation method that enables denoiser training in a human-readable RGB space by transforming the downscaled raw images to the style of a given RGB noisy image.
The noise and ISP estimations utilize rich augmentation to synthesize image pairs for denoiser training.
Experiments show that NERDS can accurately train
CNN-based denoisers ( e.g. , DnCNN, ResNet-style network) outperforming previous noise-synthesis-based and self-supervision-based denoisers in real datasets.
I NTRODUCTION
Image denoising is a conventional machine learning problem restoring original colors and patterns from noisy images.
Deep-learning-based approaches have achieved breakthroughs in recent decades due to the power of neural networks.
Early works (Zhang et al., 2017; Mao et al., 2016; Tai et al., 2017) have successfully removed additive white Gaussian noise (AWGN), which allows network training under supervision by synthesizing noisy-clean image pairs.
Nevertheless, denoising images captured by smartphone/digital cameras poses an obstacle, as it is difficult to obtain clean images for noisy images with pixel-level alignment.
Several works (Abdelhamed et al., 2018; Brummer & De Vleeschouwer, 2019) constructed datasets with the noisy-clean pairs for real-world images.
Using these pairs (Figure 1(a)), many supervised-learning-based denoisers (Yue et al., 2019; Kim et al., 2020;
Yue et al., 2020; Hu et al., 2021; Cheng et al., 2021) restore crisp images on benchmarks from the datasets.
However, constructing such datasets requires tightly controlled capturing environments, complicated post-processing, and massive human labor.
To overcome the drawback of plain supervised learning, two major types of research have been studied.
The first line of works generates realistic noisy images from clean images to utilize supervised denoiser training as visualized in Figure 1(b).
Several approaches (Chen et al., 2018; Cha et al., 2021; Hong et al., 2020; Jang et al., 2021) adopt generative models using unpaired noisy-clean images based on GAN (Goodfellow et al., 2014), but they achieve limited accuracy on real noise.
Some other works synthesize realistic noise using existing noisy-clean image pairs (Zamir et al., 2020;
Abdelhamed et al., 2019) or metadata for real cameras (Brooks et al., 2019; Guo et al., 2019), but they are limited in generalization for unseen noise.
The second category aims to learn denoisers without clean images .
The first work (Lehtinen et al., 2018) in this category proposed the learning framework using multiple noisy images.
After that, many self-supervised-learning approaches (Batson & Royer, 2019; Krull et al., 2019; Byun et al., 2021) use single noisy images (Figure 1(c)), which
[Figure]
enable easy data collection and denoiser adaptation to the test noise.
However, they are still limited in real-world applications due to the requirements of custom network architectures and strong statistical noise assumptions.
To address the above limitations on camera denoiser training, we propose a new solution pipeline, namely Noise Estimation for RGB Denoising & Synthesis (NERDS), that generates noisy-clean image pairs from raw-RGB noisy image pairs.
The pipeline composes three parts–noise estimation,
ISP estimation, and denoiser training.
We found that the pixel variance of natural clean images is robust to image downscaling, which is a widely-used denoiser for low-resolution images (Nah et al., 2019).
The noise estimation adopts a Poisson-Gaussian noise model for raw images from sensors and optimizes its noise parameters by the pixel variances of downscaled images.
The downscaled images and the estimated noise parameters enable generating pseudo-noisy and pseudo-clean image pairs at low-resolution (Figure 1(d)).
Training denoisers on human-readable RGB images from real cameras has another issue: the conversion from the raw images to the RGB images is a black box.
Our ISP estimation enables noise synthesis on the RGB space by learning RAW2RGB conversion using raw-RGB noisy image pairs 1 .
Our denoiser training can utilize rich data augmentation based on estimated noise parameters and ISPs.
Specifically, we introduce two techniques for this framework.
First, a reparametrization trick allows estimating noise parameters through a gradientdescent-based optimizer.
Second, a technique for style disentanglement from raw-RGB noisy image pairs.
We summarize our contributions as follows:
• To the best of our knowledge, this is the first work to synthesize noisy-clean RGB image pairs at low-resolution for accurate camera denoiser training from raw-RGB noisy image pairs.
• We formulate noise estimation for Poisson-Gaussian noise as an optimization problem, and a novel reparameterization trick allows to estimate accurate noise parameters through gradient-descent.
• We propose a neural network that estimates the RAW2RGB conversions (or ISPs) used for given raw-RGB noisy image pairs.
The ISP estimation generates realistic noisy-clean RGB image pairs from raw images.
• Our frameworks can train general CNN-based denoisers ( e
g. DnCNN, ResNet-style network) accurately for given test noisy images by performing noise synthesis using them.
R ELATED W ORK 2. B LIND I MAGE D ENOISING
Traditional methods Classical methods usually denoise noisy images without training data using wavelet (Donoho & Johnstone, 1995), filtering (Buades et al., 2005) including BM3D (Dabov et al., 2007), optimization (Elad & Aharon, 2006; Mairal et al., 2009; Gu et al., 2014), and effective prior (Zoran & Weiss, 2011).
However, they perform limited accuracy compared to the recent deep-learning-based approaches.
Supervised-learning-based methods SIDD (Abdelhamed et al., 2018) and NIND (Brummer &
De Vleeschouwer, 2019) captured real noisy-clean image pairs to enable supervised-learning for real camera denoisers.
However, the capturing procedure is unacceptably expensive and cumbersome.
We describe the literature on realistic noise synthesis methods for the supervised-learning.
DnCNN (Zhang et al., 2017) introduced a neural network to remove additive white Gaussian noise (AWGN) for the first time.
However, Guo et al.
(Guo et al., 2019) demonstrated the limitation of
AWGN denoisers to signal-dependent or spatially-correlated noise, which are known as the characteristics of real-world images.
To alleviate this problem, two lines of works have been researched.
The first category uses generative models for noise synthesis with stable learning (Chen et al., 2018), various noise characteristics (Jang et al., 2021), knowledge distillation (Wu et al., 2020), self-supervised-learning (Cha et al., 2021), and conditional adversarial networks (Hong et al., 2020).
Nevertheless, the scene statistics mismatched between clean and noisy datasets make it difficult to train accurate denoisers in practice.
The second category investigates camera noise modeling.
CBDNet (Guo et al., 2019) transforms
AWGN into realistic noise by using signal-dependent noise parameters and simulating in-camera
ISP functions, such as gamma correction and demosaicing.
UPI (Brooks et al., 2019) converts RGB images to raw images and simulates noise using metadata of specific cameras.
CycleISP (Zamir et al., 2020) trains neural networks for both RAW2RGB and RGB2RAW conversions on large-scale datasets with specific ISP and noise settings.
Noise Flow (Abdelhamed et al., 2019) synthesizes raw noisy images using normalizing flow and metadata.
The work on (Chang et al., 2020) proposed a GAN-based framework for noise generation in raw images which is adaptive to the camera.
In (Zhang et al., 2021), the authors claimed that the noise levels in metadata are inaccurate.
Recently, the method in (Kousha et al., 2022) models the RGB noise distribution using normalizing flow.
SCUNet (Zhang et al., 2022) proposed a practical noise model for a general-purpose denoiser.
However, all the above methods require real noisy-clean image pairs, ISPs, metadata, or high-quality clean images which are not always available.
More importantly, the noise synthesis using predetermined training datasets leads to poor denoising performances on unseen noise.
In contrast, the proposed method (NERDS) synthesizes realistic noisy-clean image pairs from only noisy images, which enables accurate denoiser training specialized in the test images.
Self-supervised-learning-based methods Noise2Noise (Lehtinen et al., 2018) introduced a framework that trains denoisers using noisy images only for the first time.
Noise2Void (Krull et al., 2019), Noise2Self (Batson & Royer, 2019), Noise2Same (Xie et al., 2020), and Neighbor2Neighbor (Huang et al., 2021) adopted advanced approaches which can train denoisers with single images corrupted by the i.i.d noise.
Noisier2Noise (Moran et al., 2020), NAC (Xu et al., 2020), and R2R (Pang et al., 2021) add additive noise to the given noisy images to make auxiliary training pairs by using prior knowledge (or assumptions) of the noise distribution.
Notably, the blind-spot network (BSN) in (Krull et al., 2019) has been improved by efficient architectures with small receptive fields (Laine et al., 2019) and dilated convolutions (Wu et al., 2020).
Using the advanced BSNs, FBI-D (Byun et al., 2021) adopts a denoiser specialized in Poisson-Gaussian noise for real-world raw image denoising.
AP-BSN (Wooseok Lee, 2022) breaks the spatially-correlated noise of real-world RGB images by pixel-shuffle downsampling.
Although the above self-supervised-learning methods use practical training datasets (only noisy images), they require custom network architectures or strong noise assumptions.
In contrast, NERDS performs noise synthesis that allows training general CNN-based denoisers based on the general noise modeling for smartphone/digital cameras.
2. N OISE E STIMATION
Prior knowledge of noise distribution supports accurate restoration in most methods for image denoising, but it is not generally available in practice.
Noise estimation methods alleviate this problem, especially for the additive white Gaussian noise (AWGN) and the Poisson-Gaussian noise.
Principal component analysis (PCA) based approaches (Liu et al., 2013; Pyatykh et al., 2012; Chen et al., 2015) perform accurate AWGN estimation.
For the Poisson-Gaussian noise model, most existing methods (Abramov et al., 2010; Uss et al., 2013) including (Foi et al., 2008; Liu et al., 2014) adopt two-step approaches; estimating the local means and variances, then adopting maximum likelihood estimation (MLE) to fit the noise model.
Foi et al.
(Foi et al., 2008) proposed the Poisson-Gaussian noise model for the first time and a noise estimation algorithm based on wavelet decomposition.
Liu et al.
(Liu et al., 2014) adopted iterative patch selection for the generalized source-dependent noise.
Recently, PGE-Net (Byun et al., 2021) adopted neural networks for accurate and fast optimization based on Generalized Anscombe Transformation (GAT) (Anscombe, 1948).
In contrast, we proposed a noise estimation method for Poisson-Gaussian noise using natural scene statistics with gradient-descent-based optimization.
P RELIMINARY : N OISE M ODELING
In a digital camera, an image sensor converts light into a digital signal (or a raw image), and an image signal processor (ISP) converts it into a human-readable RGB image.
We regard that the noise of RGB images originates from the image sensor and is transformed by the ISP.
Thus, we model noise distribution of raw images and RAW2RGB conversion including ISPs.
Raw Image: Poisson-Gaussian (P-G)
Noise A common noise model for raw images follows the
Poisson-Gaussian distribution (Foi et al., 2008), defined as:
[Equation]
which is a heteroscedastic Gaussian where z is the true signal, x is a raw noisy image observed on real image sensors, and β 1 , β 2 ≥ 0 are signal-dependent and signal-independent noise parameters.
Real cameras provide x as well as β 1 and β 2 in metadata.
However, the noise parameters in metadata are often inaccurate (Zhang et al., 2021).
Section A.1.1 discusses the noise distribution of raw images further.
RGB Image:
Transformed Noise ISPs transform raw images into RGB images using nonlinear functions, such as demosaicing, white balancing, color correction, and tone mapping.
The transformation changes noise distribution (e.g., breaking the i.i.d property in equation 1).
Moreover, users can retouch image tones and colors, altering the noise distribution further.
We define such
RAW2RGB conversion ( T ) as a function of the image style ( s ),
[Equation]
where x is a raw noisy image obtained from real image sensors and y is a RGB noisy image with custom tones and colors.
The real noise datasets (Abdelhamed et al., 2018; Plotz & Roth, 2017) used a simple and open-source ISP for the entire dataset without any image retouching processes.
However, modern smartphones and digital cameras use custom ISPs with hidden internal functions.
They provide multiple image styles depending on scenes.
The following section provides a general framework to estimate P-G noise parameters ( β 1 , β 2 ) and
conversion ( T ) with image style ( s ) using only noisy images ( x , y ).
P ROPOSED M ETHOD 4. O VERVIEW
For a given noisy pair of raw ( x ) and RGB ( y ) images, the proposed method composes three steps to restore the RGB clean image without auxiliary training data.
First, we estimate noise parameters ( β 1 , β 2 ) from x (Section 4.3).
Second, we learn the conversion from x to y while disentangling image style ( s ) (Section 4.4).
Third, we synthesize diverse pairs of noisy-clean images to train arbitrary RGB denoisers with a general supervised-learning framework (Section 4.5).
[Figure]
Synthesis
[Figure]
Noise Synthesis
[Figure]
Noisy
Clean
Noisy
Clean
Noisy Noisy
Clean Clean
Noisy
Downscaling
[Figure]
Void
[Figure]
4. O BSERVATION
[Figure]
We first illustrate our observation on raw noisy images with downscaling.
Specifically, we investigate the statistical characteristics variances of 256 × 256 images in the validation set of
SIDD (Abdelhamed et al., 2018).
To evaluate their noise levels, we downscale all raw images for each color channel with the scaling factor of 2, and rank these images according to the differences of pixel variances before and after downscaling.
As visualized in Figure 3, we show these values in a blue curve and separate them into three levels –low, medium, high.
It is observed that the images with high differences are very noisy, while the images with low differences present clean textures.
This phenomenon indicates that the true signal z is more robust to the pixel variances through downscaling than real noise.
That is why we propose the following method, which uses the difference of pixel variances to estimate noise levels without clean images.
4. P OISSON -G AUSSIAN N OISE E STIMATION VIA A R EPARAMETERIZATION T RICK
We aim to estimate noise for x without any information other than x .
For this, we find an additive noise ( N ) for the downscaled image ( x d ) by solving the following optimization problem,
[Equation]
where N ∼ N (0 , β 21 x d + β 22 ) is the Poisson-Gaussian noise discussed in equation 1 and V ar ( · ) denotes a function that outputs the pixel variance of the input image.
We can approximate β 1 and β 2 as noise parameters for x , given that the difference of pixel variances through downscaling is correlated with noise levels, as discussed in Section 4.2.
Section A.1.2 analyzes the downscaling effect further.
However, equation 3 is difficult to optimize due to the non-differentiable process of noise sampling.
To alleviate this problem, we introduce a reparameterization trick that separates noise sampling into learnable parameters and sampling from a normal distribution.
Formally, we reformulate equation 3 as
[Equation]
where PG ( β , x d ) = ( β 2 1 x d + β 2 2 ) 0 .
5 denotes a Poisson-Gaussian converter and n ∼ N (0 , 1) is additive noise with a normal distribution.
Given that the only learnable parameters in equation 4 are β 1 and β 2 , a gradient-descent optimizer ( e.g. , ADAM (Kingma & Ba, 2015)) easily finds the noise parameters.
Figure 2 visualizes the optimization problem used as noise estimation for x .
We are surprised that such a simple optimization problem estimates accurate noise parameters, while downscaled images can contain remaining noise or over-smoothing textures.
This result is because
[Figure]
the well-designed optimizer learns the characteristics of signal-dependent and signal-independent noise the entire image content.
So far we have discussed noise estimation and synthesis on raw images.
For more practical applications, the following section describes RGB noise synthesis by estimating RAW2RGB conversion.
4. ISP E STIMATION WITH S TYLE D ISENTANGLEMENT
We aim to learn a RAW2RGB conversion (or an ISP) from x to y .
However, a na ` ıve network training from x to y can easily overfit to the image contents.
Moreover, to generate multiple styles of RGB images, training a neural network for each style is resource intensive.
For this, we disentangle the styles from RGB images, allowing a network to learn multiple-style generation.
Specifically, our ISP estimation composes two networks, the style encoder ( E ) and the RAW2RGB converter ( T ) (Figure 4).
E identifies the image style as a style parameter ( s ) from the raw noisy image ( x ) and the RGB noisy image ( y ) while T uses x and s as input to generate an RGB image ( ˆ y ).
Then, we can recall the equation equation 2 as
[Equation]
where s = E ( x , y ) while training E and T to minimize L1 differences between ˆ y and y .
E composes 6 residual blocks, global pooling, and a fully connected layer and T composes 4 residual blocks and parameterized ISP functions (Kim & Lee, 2022).
We use the residual blocks of two convolutional layers and one ReLU activation with 64 filters and 3 × 3 kernels.
The training data can compose a single pair or multiple pairs of raw-RGB noisy images.
For accurate RAW2RGB conversion for downscaled images, which are unseen in training, we first adopt image scale augmentation ( SA ) that changes image resolution.
SA generates multiple training patches for a style (or a patch) by randomly sampled scaling factors to prevent overfitting to a few training data.
Formally, we redefine the style parameter ( s ) as follows,
[Equation]
Second, we design a bottleneck structure that lowers the dimension of s to avoid encoding of information about image contents.
E reduces channels and resolutions of s ( s ∈ R 3 × 1 × 1 in this paper).
s conditions T by channel attention and ISP parameter generation via fully connected layers.
4. D ENOISER T RAINING WITH D ATA A UGMENTATION
We synthesize noisy-clean RGB image pairs at low-resolution to train general RGB denoisers.
The pseudo-noisy/pseudo-clean images are the downscaled raw image ( x d ) with/without additive noise
N transformed to RGB space using T .
Formally, the denoiser ( D ) training minimizes the following objective function,
[Equation]
where s is the style parameter in equation 6.
Our denoising framework allows rich augmentation for data synthesis.
First, scaling and intensity augmentation ( SIA ) increases content diversity by changing the image resolution and pixel values from 0.5 × to 1.5 × .
Second, we randomly scale noise parameters, β 1 and β 2 , from 0.5 × to 1.5 × .
This augmentation alleviates the noise estimation error in Section 4.3.
Lastly, when multiple noisy images are given, we can augment the noise parameters and the style parameters across different images.
Overall, the objective function of our denoiser training equation 7 becomes
[Equation]
where SIA is the scale and intensity augmentation, s ′ is the augmented style parameter, and N ′ is the additive noise with augmented noise parameters.
At testing, the denoiser takes the RGB noisy image ( y ) as input, just like conventional CNN-based approaches.
E XPERIMENTS 5. I MPLEMENTATION D ETAILS
Downscaling We downscale a raw image x to x d by bicubic interpolation after asymmetric 2D
Gaussian blurring with the kernel size of 21 × 21 .
We randomly select the standard deviation of the
Gaussian blur from 0 .
25 × ds to 0 .
75 × ds for each dimension, where ds denotes downscaling factor randomly selected in the range of [1 . 5 , 2 . 5] .
We use the same hyper-parameters for noise estimation,
ISP estimation, and denoiser training unless otherwise specified.
Optimization We use ADAM (Kingma & Ba, 2015), 128 × 128 image patches, and a batch size of 64 for all experiments.
Noise estimation adopts 2 .
5 × 10 4 iterations with the initial learning rate of 1 × 10 − 4 which becomes a tenth part in every 5 × 10 3 iterations.
We use the same initial learning rates for ISP estimation and denoiser training for 5 × 10 5 iterations without the learning rate decay.
Denoiser We use two simple networks as denoiser to present the generalizability of the proposed training scheme.
Specifically, NERDS+DnCNN uses DnCNN (Zhang et al., 2017) and NERDS+D uses a ResNet-style architecture, composing a global skip connection and 32 residual blocks.
Each block has two convolutional layers and one ReLU activation with 64 filters and 3 × 3 kernels.
Dataset We use BSD68 (Martin et al., 2001) which consists of 68 gray-scale images to evaluate the performance of noise estimation.
For real image denoising, we use raw noisy and RGB noisy images on SIDD (Abdelhamed et al., 2018), DND (Plotz & Roth, 2017), and MIT-Adobe
FiveK (Bychkovsky et al., 2011).
SIDD (Abdelhamed et al., 2018) consists of training, validation, and benchmark datasets.
We use 1,280 256 × 256 patches from 40 noisy images on the benchmark for denoiser training.
In the setting of extra images, we use 50 noisy images on the training dataset with low ISO levels of 100 and downscale the noisy images with the scaling factor from 1.1 to 1.2.
For DND (Plotz & Roth, 2017), we use 1,000 512 × 512 patches from 50 noisy images on the benchmark.
We evaluate the denoising performances on the benchmarks by submitting the results to the public websites for SIDD and DND.
MIT-Adobe FiveK (Bychkovsky et al., 2011) consists of 5, raw images and paired RGB images retouched by 5 photographers.
Each RGB image has its own
RAW2RGB conversion using Adobe Lightroom while the raw image contains real camera noise.
We demonstrate an extreme scenario where only 5 noisy images are available for denoiser training.
5. R ESULTS FOR N OISE E STIMATION AND S YNTHESIS
We first validate our noise estimation on additive Poisson-Gaussian noise to images on BSD68 (Martin et al., 2001), and then visualize the noise synthesis at low-resolution from noisy images on MITAdobe FiveK (Bychkovsky et al., 2011).
Poisson-Gaussian noise estimation We demonstrate the effectiveness of our NERDS-raw on P-G noise estimation by comparing with Foi et al.
(Foi et al., 2008), Liu et al.
(Liu et al., 2014), and PGENet (Byun et al., 2021) which estimate P-G noise parameters from noisy images.
Table 1 presents the average of the noise parameters estimated from each image on BSD68 with four different noise levels.
In most cases, NERDS-raw estimates the most accurate noise parameters.
Noisy-clean RGB image synthesis Figure 5 shows noise synthesis results using NERDS.
Each real image from MIT-Adobe FiveK has its own RAW2RGB conversion.
NERDS-noisy contains
[Table]
[Figure]
realistic noise specialized to each real image, such as blue/dark noise in dark areas with none i.i.d characteristics.
To the best of our knowledge, NERDS is the first work to generate the realistic noisy images and the paired clean images for the unknown RAW2RGB conversions (or ISPs) by using only raw-RGB noisy image pairs.
Existing works for noise synthesis (Brooks et al., 2019; Zhang et al., 2022; Zamir et al., 2020;
Abdelhamed et al., 2019; Kousha et al., 2022) require real noisyclean image pairs, metadata, and ISPs.
Thus, na ` ıve comparisons between NERDS and the existing methods are unfair, but we present them in Section A.2.
5. C OMPARISONS TO THE S OT A D ENOISING M ETHODS
Results on benchmarks Given that NERDS trains denoisers without clean images, we compare our denoisers with the SotA denoising methods that do not use the pairs of real noisy-clean images .
Specifically, we compare GAT+BM3D (Dabov et al., 2007), N2V (Krull et al., 2019),
AP-BSN (Wooseok Lee, 2022), FBI-D (Byun et al., 2021),
C2N+DIDN (Jang et al., 2021), and
SCUNet (Zhang et al., 2022) on SIDD and DND benchmarks.
Table 2 presents the effectiveness of the proposed denoisers, NERDS+DnCNN and NERDS+D. C2N+DIDN (Jang et al., 2021) synthesizes realistic noise using unpaired noisy-clean images, while our NERDS+DnCNN outperforms it with the simpler denoiser (DIDN vs. DnCNN).
SCUNet (Zhang et al., 2022) synthesizes realistic noise using high-quality clean images and the predetermined noise models including specific noise levels and ISP pipelines.
The other works require only noisy images via prior-based filtering (Dabov et al., 2007), self-supervised learning (Krull et al., 2019; Wooseok Lee, 2022; Wu et al., 2020), and noise estimation (Byun et al., 2021).
The works without clean images perform denoising in raw image space and received the results in RGB image space by submitting the denoised raw images to the public websites (Abdelhamed et al., 2018; Plotz & Roth, 2017).
They often fail to remove noise in RGB image space, given that they assume strong noise characteristics such as Poisson-Gaussian distributions and the i.i.d property that do not hold in the RGB image space.
Section A.1.3 presents a generalization test for NERDS+D with latency analysis to denoise a test image from scratch.
Section A.2 and A.3 provide more qualitative results and the comparisons with the noise synthesis methods which use the pairs of real noisy-clean images.
Ablation Study Our NERDS enables rich and effective augmentation for denoiser training.
Table 3 demonstrates the ablation study on the data augmentation.
The setting without image scale augmentation (Table 3(1)) uses a fixed downscaling factor of 2.
Each component improves the
[Table] [Table]
[Figure]
restoration performances. In particular, noise parameter augmentation improves over than 0.6 dB, given that it can alleviate noise estimation error.
The usage of extra images enables high-quality clean image synthesis by downscaling images with low noise levels and small scaling factors.
Results on MIT-Above FiveK We evaluate CycleISP (Zamir et al., 2020), SCUNet (Zhang et al., 2022), and NERDS+D on retouched images from MIT-Adobe FiveK (Bychkovsky et al., 2011).
CycleISP (Zamir et al., 2020) employs supervised learning on SIDD training dataset with additional synthetic data using the predetermined noise levels and ISP pipeline.
Nevertheless, CycleISP fails to denoise the retouched image as visualized in Figure 6.
SCUNet (Zhang et al., 2022) is trained on high-quality images with practically designed additive noise, but SCUNet often generates oversmooth textures.
These results are due to different noise distributions between the training images and the test images.
In contrast, NERDS+D removes severe real noise while maintaining image details.
We report more qualitative results in Figure 12.
C ONCLUSION
We present a general framework to train denoisers from noisy images, called NERDS.
The framework composes noise estimation, ISP estimation, and denoiser training.
For noise synthesis, we estimate Poisson-Gaussian noise in raw images and ISP (or RAW2RGB conversion) for each RGB image.
NERDS allows rich data augmentation for accurate denoiser training.
Experimental results show the state-of-the-art restoration accuracy on real noise benchmarks.
A A PPENDIX A. D ISCUSSIONS A.1. N OISE D ISTRIBUTION OF R AW I MAGES
Our NERDS assumes that the raw images have the noise that follows Poisson-Gaussian distribution.
However, the image sensor of a smartphone or a digital camera is a black box.
We do not know what kind of post-processing has been applied to the raw images or whether the noise level (parameters) in the metadata represent the proper parameters of Poisson-Gaussian distribution.
For instance, one of the noise levels in the metadata of Galaxy S6 ( β 2 in Figure 7) equals zero for all images which is theoretically impossible for both shot and read noise parameters of the Poisson-Gaussian model.
Nonetheless, Figure 7 presents a similar tendency between the noise levels in the metadata ( β 1 , β 2 ) and the estimated noise parameters ( ˆ β 1 , ˆ β 2 ).
Thus, we approximate the noise distribution in raw images as Poisson-Gaussian noise and show that NERDS+D can achieve satisfactory denoising results using the estimated Poisson-Gaussian noise. [Figure] A.1. C LEAN I MAGE VIA D OWNSCALING The optimal clean image via downscaling is a noise-free low-resolution image that has the same statistics as the true signal (clean high-resolution image).
Empirically, we regard the raw images downscaled after burring (low-pass filtering)
as raw pseudo -clean images.
However, blurring images breaks the optimal setting when image structures are too small ( e.g. , 1-pixel dots) or the noise is too severe compared to the size of the blur.
Figure 8 shows the evidence and limitations of the utility of pseudo-clean images used in NERDS.
First, the images downscaled without pre-blurring have similar statistics to those of the images before downscaling.
This indicates that the image scaling can be regarded as capturing an image at different distances between an object and a camera.
Second, the blurring operation reduces the std values of noisy images drastically more than clean images.
This is why we use downscaled images as pseudo-clean images.
However, blurring also reduces the std values of clean images, making it challenging to find optimal clean images.
To alleviate this difficulty, we estimate the noise by gradient-descent-based optimization and adopt augmentation of noise parameter scaling ( × 0.5 ∼ × 1.5) for accurate denoiser training. ISO
[Figure]
We further analyze the effectiveness of blur kernels by measuring KL divergence between synthesized noisy images and real noisy images.
The lower value, the better.
Table 4 visualizes ablation study on the size of blur kernels on SIDD dataset.
While noise estimation w/ clean image perform the upper bound score, noise estimation w/ blur achieves comparable performances to noise estimation w/ clean image.
In both cases of w/o blur and w/ large blur, KL divergence scores increase than noise estimation w/ blur.
These results indicate that the downscaled images work as clean images well with the proper size of blur kernels.
[Table]
A.1. E FFICIENT I NFERENCE AT T EST T IME
The proposed method composes three steps (noise estimation, ISP estimation, and denoiser training) and an additional step of denoiser testing.
Each step described above has the latency described in
Table 5.
We use GeForce RTX 2080 Ti GPU and an HD image for testing.
[Table]
The noise estimation, ISP estimation, and denoiser training using noisy test images are timeconsuming.
To skip the processes at test time, Table 6 presents a generalization test for NERDS+D trained on different datasets.
NERDS+D trained on DND has accurate restoration performance on DND but performs 2 dB lower PSNR on SIDD than the model trained on SIDD.
In contrast,
NERDS+D trained on SIDD performs accurate restoration on both datasets.
This phenomenon indicates that SIDD contains noise distributions similar to DND and that well-designed noisy images enable generalized denoiser training.
For instance, camera manufacturers can collect training datasets of only noisy images concerning the image sensor, ISP, and expected image retouching.
[Table]
A. C OMPARISONS TO N OISE S YNTHESIS M ETHODS USING R EAL N OISY /C LEAN I MAGE P AIRS
The recently proposed methods (Abdelhamed et al., 2019; Chang et al., 2020; Zhang et al., 2021; Kousha et al., 2022) synthesise noisy images with novel noise models.
These methods use noise/clean image pairs and metadata that are not accessible at test time, such as the DND dataset.
Although the proposed method is a general framework for denoising training without a clean image, we present comparisons with these methods by evaluating the noise estimation results with denoising performance and KL divergence.
Denoising performance.
Table 7 presents denoising performances for raw images on the SIDD benchmark, where we use DnCNN as the denoiser.
Although NERDS does not use clean images for noise estimation, noise synthesis, and denoiser training, NERDS achieves the best PSNR score.
Table 8 presents denoising performances for RGB images on the SIDD benchmark, where all methods use DnCNN as the denoiser.
NERDS achieves the best PSNR/SSIM scores.
When converting the denoised raw images using NERDS to RGB images, the PSNR is 35.74 dB which is lower than
RGB image denoising (36.42 dB).
[Table] [Table]
KL divergence.
Table 9 presents the comparisons of KL divergence ( D KL ) between synthesized noisy images and real noisy images.
The lower values, the better.
The proposed method outperforms all compared methods when using clean images.
We could not reproduce the results of RGB Noise
Flow (Kousha et al., 2022) since the source code was not available.
The score reported in the paper is 0.044 for RGB Noise Flow, where Noise Flow scores 0.198.
The scores of KL divergence are dependent on hyperparameters.
[Table]
[Figure]
Figure 9 visualizes noise synthesis results using NERDS.
NERDS-clean contains low-level noise while NERDS-noisy presents severe noise similar to the real noisy images.
We have empirically found that denoisers do not learn to remove noise in NERDS-clean.
This phenomenon is because the blurring and downscaling process in NERDS distorts the noise of raw images while denoisers learn to remove Poisson-Gaussian noise in the raw images.
Moreover, given that NERDS does not require any clean data, NERDS successfully synthesizes noisy-clean image pairs on DND and
MIT-Adobe FiveK datasets which are not available to access clean images.
A.3. D ENOISED I MAGES
We present more images denoised by NERDS+D on SIDD (Figure 10), DND (Figure 11), and
MIT-Adove FiveK (Figure 12).
While AP-BSN (Wooseok Lee, 2022), FBI-D (Byun et al., 2021),
SCUNet (Zhang et al., 2022), C2N (Jang et al.
, 2021)+DIDN (Yu et al., 2019), and CycleISP (Zamir et al., 2020) often fail to restore image patterns in GT, NERDS+D successfully recover the original structures.
AP-BSN uses pixel shuffle for self-supervised learning to decorrelate spatially but also loses spatial information for denoising.
FBI-D learns raw image denoising for Poisson-Gaussian noise, but denoising raw images is an indirect approach for the human visual system (or RGB space).
SCUNet over-smooth or over-sharpen images.
This restoration style helps to generate readable characters, but it can distort the patterns in GT (See Figure 10).
C2N+DIDN is the denoiser trained by generating noisy images from clean images using both noisy and clean images on SIDD datasets.
For DND benchmarks, C2N+DIDN sometimes generates artifacts or noise that significantly drops the restoration accuracy (See Figure 11).
In contrast, NERDS+D uses only noisy images on DND benchmarks to generate noisy-clean image pairs for accurate denoiser training.
CycleISP is the denoiser trained by paired noisy-clean images on SIDD datasets and paired images synthesized by the metadata of DND benchmarks.
CycleISP performs the highest PSNR and SSIM on both SIDD and DND benchmarks compared to the methods in Table 2.
However, CycleISP fails to remove the noise on MIT-Adobe FiveK (See Figure 12).
This failure comes from the noise distribution of RGB images mismatched between MIT-Adobe FiveK and SIDD (or DND) datasets.
[Figure]
nv ap-BSN
D-BSN
FBI-D
C2N+DIDN CycleISP SCUNet
NERDS+D sidd v x o o v x o o sidd b
x
v o v x o o dnd
x
o o v
o o v o fivek
v o v o o o nv ap-BSN
D-BSN
FBI-D
C2N+DIDN CycleISP SCUNet
NERDS+D sidd vo - o v x o o sidd b - - - - v - - o dndo - o o - - o
[Figure]
Input
[Table]
nv ap-BSN
D-BSN
FBI-D
C2N+DIDN CycleISP
SCUNet
NERDS+D sidd v x o o v x o o sidd b
x
v o v x o o dnd
x
o o v
o o v o fivek
v o v o o o nv ap-BSN
D-BSN
FBI-D
C2N+DIDN CycleISP
SCUNet
NERDS+D sidd vo - o v x o o sidd b - - - - v - - o dndo - fivek - - o v o
[Table]
GT AP
[Table]
FBI
[Table]
validation
[Table] [Table]
[Table]
psnr
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
0.9939_output
[Figure]
Input nv ap-BSN
D-BSN
FBI-D
C2N+DIDN CycleISP
SCUNet
NERDS+D sidd v x o o v x o o sidd b
x
v o v x o o dnd v nv ap-BSN
D-BSN
D
SCUNet
D sidd vo - o v x o o sidd b - - - - v - - o dndo - o o - - o fivek - - o v o o o 30.76/
[Table]
[Table] [Table]
8491 26.49/- ∞/1.000 PSNR/SSIM
GT
AP-BSN [36]
FBI-D [8]
C2N
[19]+DIDN [39] NERDS+D (Ours)
[Table]
[Table]
PSNR sRGB: 33.
SSIM sRGB: 0.
PSNR sRGB: 31.
SSIM sRGB
[Table]
PSNR sRGB: 30.
SSIM sRGB: 0.
PSNR sRGB: 28.
SSIM sRGB: 0.
PSNR sRGB: 32.
SSIM sRGB: 0.
PSNR sRGB: 28.
SSIM sRGB: 0.
PSNR sRGB: 33.
SSIM sRGB: 0.
PSNR sRGB: 31.
SSIM sRGB: 0.
[Table] [Table]
[Figure]
Input nv ap-BSN
D-BSN
FBI-D
C2N+DIDN CycleISP
SCUNet
NERDS+D sidd v x o o v x o o sidd b
x
v o v x o o dnd
x
o o v
o o v o fivek
v o v o o o nv ap-BSN
D-BSN
FBI-D
C2N+DIDN CycleISP
SCUNet
NERDS+D sidd vo - o v x o o sidd b - - - - v - - o dndo - o o - - o
[Figure]
FBI-D [8]
[Figure]