Robust Deep Ensemble Method for Real-world Image Denoising

Recently, deep learning-based image denoising methods have achieved promising performance on test data with the same distribution as training set, where various denoising models based on synthetic or collected real-world training data have been learned. However, when handling real-world noisy images, the denoising performance is still limited. In this paper, we propose a simple yet effective Bayesian deep ensemble (BDE) method for real-world image denoising, where several representative deep denoisers pre-trained with various training data settings can be fused to improve robustness for real-world noisy images. The foundation of BDE is that real-world image noises are highly signal-dependent, and heterogeneous noises in a real-world noisy image can be separately handled by different denoisers. In particular, we take well-trained CBDNet, NBNet, HINet, Uformer and GMSNet into denoiser pool, and a U-Net is adopted to predict pixel-wise weighting maps to fuse these denoisers. Instead of solely learning pixel-wise weighting maps, Bayesian deep learning strategy is introduced to predict weighting uncertainty as well as weighting map, by which prediction variance can be modeled for improving robustness on real-world noisy images. Extensive experiments have shown that real-world noises can be better removed by fusing existing denoisers instead of training a big denoiser with expensive cost. On DND dataset, our BDE achieves +0.28dB PSNR gain over the state-of-the-art denoising method. Moreover, we note that our BDE denoiser based on different Gaussian noise levels outperforms state-of-the-art CBDNet when applying to real-world noisy images. Furthermore, our BDE can be extended to other image restoration tasks, including image deblurring, image deraining and single image super-resolution. On benchmark datasets, our method achieves +0.30dB, +0.18dB and +0.12dB PSNR gains for image deblurring, image deraining and single image super-resolution, respectively.

Image denoising, ensemble learning, Bayesian learning

Introduction
============

Image denoising, aiming to recover latent clean images from noisy images, is a fundamental task in image processing and low-level computer vision field. In recent years, deep learning-based denoising methods flourish in this field and achieve state-of-the-art denoising performance. Especially, for additive-white-Gaussian-noise (AWGN) removal, a basic testing bed, convolutional neural network (CNN) is successfully developed for achieving superior denoising performance @zhang2017beyond over traditional denoising methods@dabov2007image [@nasri2009image; @zhang2014adaptive; @li2016image]. Subsequently, various CNN-based denoisers  @mao2016image [@tai2017memnet; @zhang2018ffdnet; @liu2018multi; @peng2019dilated; @zhang2020residual; @liu2018non; @wang2021deep; @cheng2021nbnet; @Zheng_2021_CVPR] have been studied for boosting quantitative AWGN denoising performance. However, these CNN-based Gaussian denoisers usually perform limited, when handling real-world noisy images, due to the distribution gap between Gaussian noises and real-world noises. Moreover, as shown in Fig. \[fig:noisyimage\], noises in real-world noisy image are signal-dependent, and it is more difficult to remove heterogeneous noises for one denoiser. In @zhang2018ffdnet, FFDNet is suggested to adopt different noise reduction intensities for region-aware denoising. However, FFDNet needs to provide spatially variant noise map, and is still developed for Gaussian denoising.

![Heterogeneity distribution of noises in real-world noisy image. ](0001_noisy.pdf){width="47.50000%"}

\[fig:noisyimage\]

To remove real-world noises, supervised denoising methods can be categorized into two groups in terms of training pairs, i.e., synthesizing realistic noisy images @guo2019toward [@chen2018learning; @brooks2019unprocessing; @wang2020practical; @zhu2016noise; @liu2007automatic] and collecting real-world noisy/clean pairs @SIDD_2018_CVPR. These methods have achieved promising performance on test data with the same distribution as the training set. However, they usually suffer from notable performance declines when handling real-world noisy images with unknown noise distribution. In @song2020grouped, GMSNet is trained on both real-world and synthetic noisy/clean pairs to obtain better generalization performance. Nonetheless, its denoising performance is still limited for real-world noisy images with heterogeneous noise distribution. As a summary, it is hard for a single denoiser to obtain satisfying denoising performance for spatially heterogeneous noisy images.

\[!htbp\]

In this paper, we propose a simple yet effective Bayesian deep ensemble (BDE) method for real-world image denoising, where several representative deep denoisers pre-trained with various training data settings can be fused to improve robustness for real-world noisy images. The foundation of BDE is that real-world image noises are highly signal-dependent, and heterogeneous noises in a real-world noisy image can be separately handled by different denoisers. In particular, we take well-trained CBDNet @guo2019toward, NBNet @cheng2021nbnet, HINet @chen2021hinet, Uformer @wang2021uformer, and GMSNet @song2020grouped as our denoiser pool, and a U-Net is adopted to predict pixel-wise weighting maps to fuse these denoisers. Instead of solely learning pixel-wise weighting maps, Bayesian deep learning strategy is introduced to predict weighting uncertainty as well as weighting map, by which prediction variance can be modeled for improving robustness on real-world noisy images. As shown in Fig. \[fig:sel\], our BDE outperforms individual state-of-the-art denoisers.

Extensive experiments have shown that real-world noises can be better removed by fusing existing denoisers instead of training a big denoiser with expensive cost. We show the effectiveness and efficiency of our method on two real-world benchmarks, *i.e.* Darmstadt noise dataset (DND) @plotz2017benchmarking and smartphone image denoising dataset (SIDD) @SIDD_2018_CVPR. Note that the DND benchmark is taken as the main benchmark, due to none of the corresponding training set and ground truth of testing images. On DND dataset, our BDE achieves +0.28 dB PSNR gain over the state-of-the-art denoising method. One may notice that there is an ensemble method for image denoising @choi2019optimal, where CSNet is proposed to combine five denoisers with different noise levels. Our BDE is distinctive with CSNet from two perspectives: (i) CSNet is developed for gray-style Gaussian denoising task, while our BDE aims to fuse denoisers for removing real-world noises. (ii) CSNet adopts an image-level ensemble strategy, by which pre-trained denoisers cannot be well exploited and fused. In our BDE, pixel-wise ensemble strategy benefits heterogeneous noise distribution. As a support, BDE denoiser based on different Gaussian noise levels outperforms state-of-the-art CBDNet when applying to real-world noisy images.

Furthermore, our BDE can be extended to other image restoration tasks, including image deblurring, image deraining and single image super-resolution, where the proposed ensemble method can be directly applied to fuse representative restoration models for each task. On benchmark datasets, our method achieves +0.30dB, +0.18dB and +0.12dB PSNR gains for image deblurring, image deraining and single image super-resolution, respectively.

The contribution of this paper can be summarized from three perspectives.

-   A robust ensemble method is proposed to handle heterogeneity distribution in real-world noisy images, where well-trained denoisers can be fused in a pixel-wise fusion manner.

-   Bayesian deep ensemble method is adopted to predict weighting uncertainty as well as weighting map, by which prediction variance can be modeled for improving denoising robustness. On both DND and SIDD benchmarks, our method achieves better results than state-of-the-art denoisers.

-   Our ensemble method can be easily extended to other restoration tasks including image deblurring, image deraining and single image super-resolution, where notable gains are achieved on 14 benchmark datasets.

The remainder of this paper is organized as follows: In Section \[secII\], we review image denoising and ensemble deep learning methods. In Section \[secIII\], we give a detailed introduction of our robust ensemble method for image denoising, as well as its extension to other image restoration tasks. In Section \[secIV\], we conduct experiments to evaluate our method on both real-world noisy benchmarks and other image restoration tasks, in comparison to corresponding state-of-the-art methods. Finally, some conclusion remarks are given in Section \[secV\].

Related Works {#secII}
=============

Image denoising
---------------

Nowadays, most image denoising methods focus on CNN-based methods since DnCNN @zhang2017beyond, which achieve better performance than traditional methods @dabov2007image. According to the training set, we mainly review two types of denoisers, *i.e.* training on the synthetic dataset and real-world dataset.

### Image Denoising with Synthetic Dataset

The CNN-based image denoising can be dated back to 2009 @jain2008natural, however, its performance could only compare to BM3D @dabov2007image. Unfortunately, the successive denoisers in @xie2012image [@burger2012image] also do not achieve success. The milestone of image denoising is DnCNN @zhang2017beyond, which combines residual learning stage and batch normalization (BN) @ioffe2015batch and achieves the state-of-the-art performance. Subsequently, some methods, such as REDNet @mao2016image, MemNet @tai2017memnet, MWCNN @liu2018multi, DSNet @peng2019dilated, RDN @zhang2020residual, and DCDicL @Zheng_2021_CVPR propose a more complex architecture to obtain performance progress. Instead of designing the network architecture, some methods utilize some image priors, such as non-local information and sparse representation, to promote the denoising performance. In N3Net @plotz2018neural and NLRN @liu2018non, they exploit the non-local image prior by proposing non-local modules for noise removal. While Cheng *et al.* @cheng2021nbnet propose a non-local attention module to explicitly learn the basis generation as well as subspace projection. Wang *et al.* @wang2021deep incorporate the global non-linear smoothness constraint prior to CNN which can improve the interpretability of deep learning. To train controllable denoiser, @zhang2018ffdnet [@du2020flexible; @zhang2018learning] take the noisy image with its noise map as input. However, those methods are trained and tested on Gaussian noise, and most of them are less effectiveness even useless when they are applied to real-world noisy images due to the huge domain gap.

Referring to the camera image signal processing (ISP) pipeline, more complex noise can be synthesized for removing the real-world noisy images @guo2019toward [@zamir2020cycleisp]. Some methods @chen2018learning [@brooks2019unprocessing; @wang2020practical] even process the noisy images in the raw domain, which directly read out from the sensor in the raw Bayer format before demosaicing and other ISP-processing, for better denoising performance. However, the denoising performance of those denoisers on a real noisy dataset relies heavily on the similarity between image priors and noise statistics. While the domain gap between synthetic noise and real noise is still non-negligible. For example, the denoising performance of CBDNet @guo2019toward on DND and SIDD is much weaker than Uformer @wang2021uformer which is trained on real-world dataset.

### Real-world denoisers

To remove real-world noise, the simple and effective approach is collecting real-world noisy/clean pairs, such as SIDD @SIDD_2018_CVPR. Then numerous denoisers, such as MPRNet @mehri2021mprnet, MIRNet @zamir2020learning, NBNet @cheng2021nbnet, Uformer @wang2021uformer, and HINet @chen2021hinet, are trained only SIDD dataset and achieve impressive denoising performance on both SIDD and DND benchmarks. However, collecting real-world noisy/clean image pairs is extremely challenging and non-trivial. Those denoisers also fall into overfitting the training set and lack generalization. Training on both synthetic dataset and real-world dataset can alleviate the overfitting problem @anwar2019real [@zamir2020cycleisp; @jang2020dual; @song2020grouped]. GMSNet @song2020grouped achieves impressive denoising performance on the DND benchmark, which has no corresponding training set. Recently, the unsupervised and self-supervised learning are introduced for real-world image denoising because of no training pairs  @ulyanov2018deep [@lehtinen2018noise2noise; @krull2019noise2void; @batson2019noise2self; @quan2020self2self; @moran2020noisier2noise; @xu2020noisy; @byun2021fbi; @laine2019high; @wu2020unpaired; @huang2021neighbor2neighbor]. Those methods often design specific architecture, such as blind-spot mechanism @krull2019noise2void, to learn self-information for the noisy image itself. Naturally, the domain gap is neglected to obtain generalization. However, the denoising performance of those denoisers is still far less than the supervised learning, and they are often very time-consuming.

Bayesian deep learning
----------------------

In Bayesian deep learning, input-dependent aleatoric and epistemic uncertainty are two main types of uncertainty. The input-dependent aleatoric uncertainty, which is also known as heteroscedastic uncertainty, measures the inherent noise in observation data and can't be reduced by providing more data. Monte Carlo dropout @gal2016dropout, which utilizes the dropout layer before sending to the convolution layer, is often adopted to simulate aleatoric uncertainty. The epistemic uncertainty models the uncertainty in the model parameters due to limited data and knowledge. To capture both of them, Kendall *et al.* @kendall2017uncertainties propose a unified framework and improve the robustness and performance on depth regression and semantic segmentation tasks. Recently, Bayesian deep learning is introduced to many vision tasks, *e.g.* face recognition @chang2020data, cross-task @zamir2020robust, multi-source transfer learning @CHANDRA202054, classification @zhao2020probabilistic, and semantic segmentation @yeo2021robustness, to improve the robustness of CNN models.

Ensemble learning {#ensemble_model}
-----------------

Our work is an ensemble deep learning method @yeo2021robustness [@kumar2020ensemble; @hao2020visual; @pinto2021ensemble; @ganaie2021ensemble; @zhang2021new]: it can combine several pretrained methods to obtain better generalization performance, naturally, the excavated prior can be well-used for the current task. In @liu2020improved, Liu *et al.* adopt a spatial local fusion strategy to combining nonlocally centralized sparse representation (NCSR) @dong2012nonlocally and FFDNet @zhang2018ffdnet for improving Gaussian image denoising performance. Yang *et al.* @yang2020image stack several base denoisers, *i.e.* the well-established shrinkage (attenuation) based denoising method, for further performance improvement. Choi *et al.* @choi2019optimal combine several Gaussian denoisers by estimating the mean squared error (MSE) of denoised images and then estimating combining weights with convex optimization. The proposed method can well combine denoising results with accurately predicted MSE. However, if the MSE is inaccurate, it may obtain a worse result.

For other image restoration tasks, ensemble methods are also proposed for improving image restoration performance. Liao *et al.* @liao2015video propose a two-stage video super-resolution ensemble method that the first and second stages aim at merging SR drafts and final deconvolution respectively. In @liu2017robust, Liu *et al.* propose a temporal adaptive neural network that takes as input a number of aligned LR frames and applies filters of different temporal sizes to generate multiple HR frame estimates. @cho2021compression proposes an ensemble method for compression artifacts reduction by fusing different restoration methods on a feature map level.

Proposed Method
===============

 \[secIII\] In this section, we first present the motivation of our robust ensemble method, then the proposed Bayesian deep ensemble (BDE) denoising method is proposed, and finally BDE is extended to other image restoration tasks, including image deblurring, image deraining and single image super-resolution.

Problem Formulation and Motivation
----------------------------------

Given a noisy image $\mathbf{y}  $, the latent clean image $\mathbf{z}$ can be recovered by a denoiser $\mathcal{D}$ $$\begin{aligned}
 \label{eq:denoise1}
 \mathbf{\hat{z}} =   \mathcal{D} \left( \mathbf{y} \right).  \end{aligned}$$ To train the denoiser $\mathcal{D}$, massive training pairs $\{\mathbf{y}_n, \mathbf{x}_n^{gt}\}_{n=1}^N$ should be prepared, where $\mathbf{x}_n^{gt}$ is the ground-truth clean image and $N$ is the number of training pairs. Then various loss functions, e.g., $\ell_1$-norm @wang2018esrgan, mean square error @dong2015image [@shi2016real; @zhang2017beyond], perceptual loss @johnson2016perceptual and adversarial loss @ledig2017photo, can be adopted to learn the parameters of denoiser $\mathcal{D}$ by closing the distance between denoised result $\hat{\mathbf{z}}_i$ and ground-truth clean image $\mathbf{x}_i^{gt}$. For real-world noisy images, training pairs can be obtained by synthesizing realistic noisy images @guo2019toward [@chen2018learning; @brooks2019unprocessing; @wang2020practical; @zhu2016noise; @liu2007automatic] and collecting real-world noisy/clean pairs @SIDD_2018_CVPR. Recently, GMSNet @song2020grouped is trained on both real-world and synthetic noisy/clean pairs to obtaining better generalization performance.

![The problem formulation: Given well-trained denoisers with different training settings, how to obtain a robust and better denoised result?](Problem.png){width="47.50000%"}

\[fig:problem\]

\[!htbp\]

\[fig:fusingmeth\]

Currently, there are several representative deep denoisers have been learned, forming a denoiser pool $\mathbb{D}=\{\mathcal{D}_c\}_{c=1}^C$, as shown in Fig. \[fig:problem\]. Considering the heterogeneity distribution of real-world noisy images, one denoiser from $\mathbb{D}$ may perform well for some regions while failing for other regions. Therefore, we in this work suggest that heterogeneous noises in a real-world noisy image can be separately handled by different denoisers. To this end, the denoised result $\hat{\mathbf{z}}_c= \mathcal{D}_c(\mathbf{y})$ can be fused by $$\begin{aligned}
 \label{eq:addres}
\mathbf{\hat{x}}  = \! \sum_{c=1}^{C}  \mathbf{w}_c \odot \hat{\mathbf{z}}_c,  \ s.t. \ \mathbf{w}_1 +  ... + \mathbf{w}_C=\mathbf{1},\end{aligned}$$ where $\mathbf{w}_c$ is the weighting map, and $\odot$ is the entry-wise product. Different from @choi2019optimal, where the image-level ensemble method is proposed for Gaussian denoising, our proposed ensemble strategy would benefit heterogeneous noise. As support, our method by fusing different Gaussian noise levels outperforms state-of-the-art CBDNet when applying to real-world noisy images. The remaining issue is how to obtain the weighting map, for which we propose a Bayesian learning method.

Bayesian Deep Ensemble Denoising Method {#fusingmethod}
---------------------------------------

As shown in Fig. \[fig:wholeall\], our pipeline for estimating the weighting maps contains two parts: 1) estimating the score for each noise removal result by the scoring network $\mathcal{F}$ with parameters $\Theta$ $$(\mathbf{s}_c, \boldsymbol{\sigma}_c) = \mathcal{F}_\Theta (\mathbf{y}, \hat{\mathbf{z}}_c),$$ and 2) obtaining the final weighting maps by the SoftMax function: $$\label{softmax}
(\mathbf{w}_c)_{ij} =  \frac{\exp (\mathbf{s}_c)_{ij}} { \sum^C_{c=1} \exp (\mathbf{s}_c)_{ij}},$$ where $\left(i,j\right)$ is the position in the scoring map, and $C$ denotes the number of fused denoisers. For the scoring network, it takes one of the denoised results $\hat{\mathbf{z}}_c$ with noisy image $\mathbf{y}$ as input and outputs two maps: the scoring map and uncertainty map $\boldsymbol{\sigma}$. It should be pointed out that the scoring network is based on Bayesian deep learning by considering both input-dependent aleatoric uncertainty and epistemic uncertainty @kendall2017uncertainties. For input-dependent aleatoric uncertainty, it comes from two folds: i) Both different networks' architectures and different training sets lead to various denoised results. ii) The unknown test set with unknown-type noise leads to completely unknown effects. Epistemic uncertainty is modeled by placing a prior distribution (*e.g.* Gaussian prior distribution) over a model's weights, and then trying to capture how much these weights vary given some data. Here, we assume a Gaussian prior distribution with zero mean over CNN's weights, and estimate the weighting uncertainty $\boldsymbol{\sigma}$ (*i.e.* variance) with the scoring map to capture how much distribution noise we have in the output scoring maps. Note that $\boldsymbol{\sigma}$ is only utilized to train the robust scoring network and abounded during the testing phase.

### Network Configuration

As shown in Fig \[fig:uncer\], our scoring network is a 4-level U-Net architecture with 3 sum skip connections: i) To model input-dependent aleatoric uncertainty, we add the dropout layer after each ReLU layer to build MC dropout @gal2016dropout. ii) All the convolutional kernel size is set to $3\times3$. iii) In the first level, the feature map is set to 32, and doubled after max-pooling. iv) Each level of the network contains two convolution layers in both encoder and decoder, and the final layer of the scoring network is 14.

### Learning Objective

Once obtaining all the weighting maps, we can get the fused results by $\mathbf{\hat{x}}= \sum_c^C \mathbf{{w}}_c \odot \hat{\mathbf{z}}_c$. And then it is natural to impose supervision based on the final fused denoising result, $$\begin{aligned}
 \label{eq:sum}
    \mathcal{L}_{fuse} = \underset{\Theta}{\min}   \| \sum_{c=1}^C \mathbf{{w}}_c \odot \hat{\mathbf{z}}_c - \mathbf{x}^{gt} \|_1.\end{aligned}$$

Besides, we suggest introducing supervision on weighting map. The ground-truth weighting map can be generated by two approaches: i) utilizing the convex optimization in @choi2019optimal; ii) selecting the best one and then setting its mask to 1 while the others to zeros. However, the first approach couldn't generate pixel-wise ground-truth weight in real time. Hence, we adopt the second approach for generating the ground-truth weights, as follows: $$\begin{aligned}
\label{maskgt}
(\mathbf{w}_c^{gt})_{ij} =\left\{
\begin{aligned}
1 & , \ \ \text{if} \ \ (\mathbf{e}_c)_{ij}= \min\{ (\mathbf{e}_c)_{ij}\}_1^C, \\
0 & , \ \ \text{otherwise},
\end{aligned}
\right.\end{aligned}$$ where $(\mathbf{e}_c)_{ij}$ is the $i,j$-th pixel MSE between $c$-th prediction result and ground-truth, *i.e.* $(\mathbf{e}_c)_{ij}= \| \left(\hat{\mathbf{z}}_c\right)_{ij} - \left(\mathbf{x}^{gt}\right)_{ij} \|^2$. Referring to @kendall2017uncertainties, we adopt the following negative log-likelihood loss $$\label{eq:nlltv}
\begin{aligned}
\mathcal{L}_{NLL} = \underset{\Theta}{\min} &\frac{1}{C} \sum_{c=1}^C  \frac{1}{{ \boldsymbol{\sigma}}^{2}_c} \| \mathbf{{w}}_c - \mathbf{w}_c^{gt} \|_2 \\&+ \log \left({\boldsymbol{\sigma}}_c \right)  + \lambda_{TV} TV\left( \mathbf{w}_c \right),
\end{aligned}$$ where $\hat{\boldsymbol{\sigma}}$ is uncertainty map, $TV\left( \cdot \right)$ denotes the Total Variation regularization to smooth the estimated weighting map and $\lambda_{TV}$ is the trade-off parameter.

Finally, we can get the total loss function for learning $\mathcal{F}_\Theta$ $$\begin{aligned}
 \label{eq:weights}
\mathcal{L} = \mathcal{L}_{NLL} + \lambda_0 \mathcal{L}_{fuse} ,\end{aligned}$$ where $\lambda_0$ controls the weighting between the two loss terms.

\[!htbp\]

\[c\]<span>0.137</span>

![image](0027_14.png){width="0.99\linewidth"}

Noisy Image

\[fig:noisya2\]

\[c\]<span>0.137</span>

![image](0027_14_CBNet_res.png){width="0.99\linewidth"}

CBDNet @guo2019toward

\[fig:CBDNet\]

\[c\]<span>0.137</span>

![image](0027_14_NBNet_res.png){width="0.99\linewidth"}

NBNet @cheng2021nbnet

\[fig:NBNet\]

\[c\]<span>0.137</span>

![image](0027_14_HITNet_res.png){width="0.99\linewidth"}

HINet @chen2021hinet

\[fig:HINet\]

\[c\]<span>0.137</span>

![image](0027_14_Uformer_res.png){width="0.99\linewidth"}

Uformer @wang2021uformer

\[fig:Uformer\]

\[c\]<span>0.137</span>

![image](0027_14_GMSNet_res.png){width="0.99\linewidth"}

GMSNet @song2020grouped

\[fig:GMSNet\]

\[c\]<span>0.137</span>

![image](0027_14_ensemble.png){width="0.99\linewidth"}

BDE

\[fig:23\]

\[c\]<span>0.137</span>

![image](v_probs.png){width="0.99\linewidth"}

\[fig:23\_43v\]

\[c\]<span>0.137</span>

![image](0027_14_CBDNet_res_probs.png){width="0.99\linewidth"}

\[fig:CBDNet34v\]

\[c\]<span>0.137</span>

![image](0027_14_NBNet_res_probs.png){width="0.99\linewidth"}

\[fig:NBNet34v\]

\[c\]<span>0.137</span>

![image](0027_14_HINet0_res_probs.png){width="0.99\linewidth"}

\[fig:HINet34v\]

\[c\]<span>0.137</span>

![image](0027_14_Uformer_res_probs.png){width="0.99\linewidth"}

\[fig:Uformer34v\]

\[c\]<span>0.137</span>

![image](0027_14_GMSNet_res_probs.png){width="0.99\linewidth"}

\[fig:GMSNet34v\]

\[c\]<span>0.137</span>

![image](v_probsR.png){width="0.99\linewidth"}

\[fig:2343v\]

\[c\]<span>0.137</span>

![image](u_probsL.png){width="0.99\linewidth"}

\[fig:23\_43u\]

\[c\]<span>0.137</span>

![image](0027_14_CBNet_res_uncertainty.png){width="0.99\linewidth"}

\[fig:CBDNet34u\]

\[c\]<span>0.137</span>

![image](0027_14_NBNet_res_uncertainty.png){width="0.99\linewidth"}

\[fig:NBNet34u\]

\[c\]<span>0.137</span>

![image](0027_14_HINet_res_uncertainty.png){width="0.99\linewidth"}

\[fig:HINet34u\]

\[c\]<span>0.137</span>

![image](0027_14_Uformer_res_uncertainty.png){width="0.99\linewidth"}

\[fig:Uformer34u\]

\[c\]<span>0.137</span>

![image](0027_14_GMSNet_res_uncertainty.png){width="0.99\linewidth"}

\[fig:GMSNet34u\]

\[c\]<span>0.137</span>

![image](u_probsR.png){width="0.99\linewidth"}

\[fig:2343u\]

\[!t\]

\[tab:dnd\]

Discussion
----------

Here, we mainly focus on analyzing our first stage, which merges different denoisers' results to obtain preliminary noise removal results. From different views, it can help to more easily understand our proposed method.

### Relationship with path-restore

In @yu2021path, Yu *et al.* restore the corrupted image by a path-restore CNN method that can dynamically select an appropriate route for each image region. The pathfinder in this path-restore CNN method can provide the thumbnail masks for selecting different features from the previous layer. As shown in Fig. \[fig:fusingmeth\], our proposed ensemble method predicts the weighting maps to select or combine different regions of denoised results. In this view, our proposed can be seen as a region-aware method that is similar to the path-restore CNN method. Different from @yu2021path, our proposed method can merge infinite denoisers while the paths in the path-restore CNN method are fixed. Moreover, our predicted probability map is a pixel-wise map instead of the thumbnail one, which can provide more detailed region-aware information.

### Relationship with dictionary learning

In dictionary learning, the restored image signal $\mathbf{\hat{x}}$ can be obtained a linear combination of atomic bases, *i.e.* $\mathbf{\hat{x}}=\mathbf{d}_{n} \mathbf{w}$, where $\mathbf{d}_{n}$ represents the dictionary atom set and $\mathbf{w}$ is the corresponding coefficient vector @zheng2021deep. If atomic bases are different denoisers, it is very similar to our ensemble method in Eq.(\[eq:addres\]), which is also a linear combination function. However, in our proposed method, the atoms, *i.e.* different denoisers, are well-trained and then fixed, the coefficient $\mathbf{w}$ can be solved by training a CNN and then fast inference in the testing phase. While $\mathbf{d}_{n}$ and $\mathbf{w}$ in dictionary learning are learned by alternating direction method of multipliers (ADMM) algorithm. During the test phase, our ensemble method can be very fast by GPU acceleration, but the dictionary learning is often extremely time-consuming. Moreover, our BDE is hopeful to fuse infinite denoisers rather than the fixed number. Above all, our BDE can be seen as an improved dictionary learning method.

Extension to Other Image Restoration Tasks
------------------------------------------

Although our BDE is proposed for image denoising, it is potential to be beneficial for other image restoration tasks. In this work, we take image deblurring, image deraining and single image super-resolution as examples. By selecting several well-trained corresponding restoration models into pool $\mathbb{D}$, the learning and inference of BDE for these tasks are exactly same with image denoising. Although degradation in these tasks may be spatially dependent, pixel-wise ensemble by our BDE is also effective, and can notably improve the performance on benchmark datasets (Referring to Sec. \[sec:experiments extension\]).

Experiments
===========

 \[secIV\] In this section, we first provide the detailed implementation of our proposed method, such as training sets and testing sets. Then different results and the ablation study of our method are displayed for demonstrating the effectiveness of our proposed BDE method. For all the compared methods, we download their source codes and well-trained models from the corresponding authors' websites. PSNR and SSIM are utilized as basic metrics for evaluating the performance of different methods. And the visual results are also provided for quantitative comparison.

\[!htbp\]

\[tab:factor\]

\[!htbp\]

\[tab:sidd\]

Implement details
-----------------

 \[sec:imp\_detail\] For real-world image denoising, DND @plotz2017benchmarking and SIDD @SIDD_2018_CVPR are the most commonly used benchmark. DND contains 50 pairs of real-world noisy images and corresponding ground truth images for evaluation. However, it doesn't provide any noisy/clean training pairs for training, and the ground truth images for evaluation are not publicly available. The PSNR and SSIM values can be obtained by the online server after submitting the noise removal results, and only 10 samples with size $512\times512$ are provided the detailed quantitative index. On the contrary, SIDD consists of both training pairs, validation pairs, and testing pairs. Here, we follow @song2020grouped to use the medium version of SIDD (320 image pairs) as the real-world training set, and the validation pairs are utilized as our testing set. Note that the images of both DND and SIDD are more than 4K resolution. To synthesize the noisy/clean image pairs, we employ the real noise model in CBDNet @guo2019toward with DIV2K dataset @Agustsson_2017_CVPR_Workshops to generate the huge number of training pairs with cropped size $256 \times 256$. The noise generating parameters are the same as CBDNet.

To train the scoring network, we employ the Adam optimizer with $\beta_1=0.9$ and $\beta_2=0.999$. The mini-batch size is set to 32, and $128\times128$ patch size is randomly cropped from the dataset during training. The default data augmentations, such as flipping, rotating, and cropping, for better generalization. Note that Mix-Up @wang2021uformer is also adopted during the training stage. The initial learning rate is set to 1e-4 and is decayed with factor 0.5 after every 200 epochs until 1e-6. Our scoring network is trained by 2000 epochs on one NVIDIA RTX 2080Ti GPU with two days. For each epoch, it only contains 100 iterations and is totally trained for 2$\times$10$^5$ iterations. The code is built on the PyTorch platform and is released at <https://github.com/lpj0/BDE>.

Comparison with State-of-the-arts
---------------------------------

### Evaluation on Real-world Noisy Images

For evaluating the effectiveness of our proposed method, we first published the results on DND and SIDD benchmarks. We collect 10 famous denoisers, *i.e.* BM3D @dabov2007image, DnCNN @zhang2017beyond CBDNet @guo2019toward, CycleISP @zamir2020cycleisp, MPRNet @mehri2021mprnet, MIRNet @zamir2020learning, NBNet @cheng2021nbnet, GMSNet @song2020grouped, Uformer @wang2021uformer, and HINet @chen2021hinet for comparison. Note that our method should first fuse different denoisers, *i.e.* taking the denoised results of different methods as input. To fuse different characteristic denoisers, we first classify the denoisers into three types in terms of training sets: i) only using the medium version of SIDD, *e.g.* NBNet, HINet, and Uformer; ii) using both the synthetic dataset and the medium version of SIDD, *e.g.* GMSNet, CycleISP. iii) mainly using the synthetic dataset, but also with other negligible real-world datasets, *e.g.* CBDNet. And the fused denoisers should contain all types to make their respective advantages complementary to each other. Then in terms of the PSNR value and class type, we select five denoisers, *i.e.* CBDNet, NBNet, HINet, Uformer, and GMSNet, as our denoiser pool.

<span>**<span>Results on DND dataset.</span>**</span> As shown in Table. \[tab:dnd\], our proposed method outperforms the state-of-the-art GMSNet 0.28 dB. Among the fused denoisers, GMSNet owns the highest performance in terms of PSNR and SSIM metrics, which is trained on both the synthetic noisy/clean pairs and the medium version of SIDD. In comparison with CBDNet, our method outperforms their 2.4dB. Even if the mean PSNR/SSIM performance of other fused denoisers, *i.e.* CBDNet, MPRNet, MIRNet, NBNet, and Uformer, are much lower than GMSNet, they can still provide useful information for reconstructing the restored image, which indicates the effectiveness of our proposed method. We also provide visual results for comparison, as shown in Fig. \[fig:dndv\]. The hot maps of weights and uncertainties are also provided for verifying the effectiveness of the proposed method. One can find that different parts of the denoised results are selected to combine. Then the final denoised image can be further improved.

We also trained a big model named "ModelH" that combines the fused denoisers and increases convolution channels to match the same Flops and parameters of the overall model. As listed in Tab. \[tab:dnd\], ModelH obtains similar PSNR to BDE on SSID but far lower on DND, which demonstrates the performance gain is not got by the increased Flops or parameters but by the fused denoisers. We provide running time and PSNR indexes in Tab. \[tab:factor\]. More than 96$\%$ of the running time is spent on the inference of the fused denoisers, while our scoring network is fast enough for predicting weighting maps.

<span>**<span>Results on SIDD dataset.</span>**</span> We also evaluate our method on SIDD as listed in Tab. \[tab:sidd\]. Note that SIDD has the corresponding training set, and most methods are trained on it to get higher PSNR/SSIM performance. Even though, our method also can get a higher PSNR value than the previous state-of-the-art method.

### Comparison with CSNet

As mentioned above, CSNet @choi2019optimal is also an ensemble method that can combine several denoisers by estimating MSE and convex optimization. However, it is only training and testing on gray-style Gaussian image denoising. To fair comparison, we also apply our ensemble method to gray-style Gaussian image denoising. The setting is kept the same with CSNet: REDNet @mao2016image is initial denoiser. Five well-trained denoisers, *i.e.* $\{ \mathbf{D}_{G}^{10},\mathbf{D}_{G}^{20},\mathbf{D}_{G}^{30},\mathbf{D}_{G}^{40},\mathbf{D}_{G}^{50}\}$ with noise level $\{10,20,30,40,50\}$, are token as the denoiser pool $\mathbb{D}$. And the noise is not clipped in our experiment. Note that we only compare the combination part, *i.e.* CSNet before booster (denoted as CSNet-B). Table. \[tab:bsd200\] lists the PSNR results of CSNet-B and our ensemble method on BSD68 dataset @martin2001database. One can find that our ensemble method BDE can achieve similar or better results than CSNet-B.

\[tab:bsd200\]

\[tab:csnetdnd\]

<span>**<span>Evaluating on real-world DND benchmark.</span>**</span> We further retrain CSNet-B with the same training set as our BDE in Sec.\[sec:imp\_detail\] and evaluate it on the real-world DND benchmark for a fair comparison. As shown in Table. \[tab:csnetdnd\], CSNet-B is only comparable with GMSNet, and far less than our BDE. We assume that estimating MSE is uncertain when it comes across an unknown benchmark. While our method can obtain stable results by introducing DBL to the scoring network.

### Fusing Gaussian Denoisers for Real-world Noisy Images

We train five color Gaussian denoisers as our denoiser pool and test our ensemble method on the DND benchmark to verify the effect of processing the unknown real-world noisy image. The architecture of those denoisers is CDnCNN @zhang2017beyond, which is a well-known denoiser that can be fast fine-tuned with the different noise levels. We denote those denoisers as $\{ \mathbf{D}_{C}^{nl}\}$ with noise level $nl=\{10,20,30,40,50\}$. We directly utilize the proposed method to combine Gaussian denoisers, denoted as BDE$_{GD}$. Then the fused result with its input noisy image is taken as training pairs for fine-tuning the best denoisers, which can be seen as knowledge distillation with self-information. We name this variant as BDE$_{GD}$+D.

Table. \[tab:combinegaussian\] lists the results on the DND benchmark. Note that CDnCNN-B is not reported by its authors, and we get this result from @zhou2020awgn. Even though combining several Gaussian denoiser, it can easily outperform CBDNet, which utilizes the complex noise synthesis pipeline to simulate real-world noise. This indicates our proposed method owns good generalization ability. Furthermore, after knowledge distillation, the performance of our method can be further improved. Therefore, we assume that the unknown type of noisy image can be handled by our ensemble method.

\[tab:combinegaussian\]

\[!htbp\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_input.png){width="0.99\linewidth"}

Blur Image

\[fig:Blur\_hide\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_MTRNN.png){width="0.99\linewidth"}

MTRNN @park2020multi

\[fig:MTRNN\_hide\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_MPRNet.png){width="0.99\linewidth"}

MPRNet @mehri2021mprnet

\[fig:MPRNet\_hide\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_MIMO.png){width="0.99\linewidth"}

MIMO-UNet+ @cho2021rethinking

\[fig:mimo\_hide\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_HINet.png){width="0.99\linewidth"}

HINet @chen2021hinet

\[fig:hinet\_hide\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_Restormer.png){width="0.99\linewidth"}

Restormer @Zamir2021Restormer

\[fig:Restormer\_hide\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_ensemble.png){width="0.99\linewidth"}

BDE

\[fig:hide\_u2em\]

\[c\]<span>0.24</span>

![image](93fromGOPR1089.MP4_GT.png){width="0.99\linewidth"}

Ground Truth

\[fig:hide\_gt\]

\[!htbp\]

\[c\]<span>0.12</span>

![image](100_rain.png){width="0.99\linewidth"}

Rainy Image

\[fig:rainy\_rain100l\_16\]

\[c\]<span>0.12</span>

![image](100_PreNet.png){width="0.99\linewidth"}

PreNet @ren2019progressive

\[fig:MTRNN\_rain100l\_16\]

\[c\]<span>0.12</span>

![image](100_MSPFN.png){width="0.99\linewidth"}

MSPFN @mehri2021mprnet

\[fig:MPRNet\_rain100l\_16\]

\[c\]<span>0.12</span>

![image](100_MPRNet.png){width="0.99\linewidth"}

MPRNet @mehri2021mprnet

\[fig:mimo\_rain100l\_16\]

\[c\]<span>0.12</span>

![image](100_HINet.png){width="0.99\linewidth"}

HINet @chen2021hinet

\[fig:hinet\_rain100l\_16\]

\[c\]<span>0.12</span>

![image](100_Restormer.png){width="0.99\linewidth"}

Restormer @Zamir2021Restormer

\[fig:Restormer\_rain100l\_16\]

\[c\]<span>0.12</span>

![image](100_u2em.png){width="0.99\linewidth"}

BDE

\[fig:rain100l\_16\_u2em\]

\[c\]<span>0.12</span>

![image](100_gt.png){width="0.99\linewidth"}

Ground Truth

\[fig:rain100l\_16\_gt\]

\[!htbp\]

\[tab:gopro\]

\[!htbp\]

\[tab:deraining\]

\[!htbp\]

\[tab:sisr\]

\[!ht\]

\[tab:vardnd\]

Extension to Other Image Restoration Tasks {#sec:experiments extension}
------------------------------------------

Our proposed BDE method can also be generalized to other image restoration tasks, *e.g.* image deblurring, image deraining, and single image super-resolution. Similar to image denoising, we also select five representative models as the model pool for training and testing. Other training settings can also refer to Sec. \[sec:imp\_detail\].

### Image deblurring.

As listed in Tab. \[tab:gopro\], seven state-of-the-art methods, *i.e.* DeblurGAN @kupyn2018deblurgan, DeblurGAN-v2 @Kupyn_2019_ICCV, MTRNN @park2020multi, MPRNet @mehri2021mprnet, MIMO-UNet+ @cho2021rethinking , HINet @chen2021hinet, and Restormer @Zamir2021Restormer, are selected for comparison. And MTRNN @park2020multi, MPRNet @mehri2021mprnet, MIMO-UNet+ @cho2021rethinking, HINet @chen2021hinet, and Restormer @Zamir2021Restormer are utilized as fused models. It can easily find that our method obtains $\sim$0.4dB gain on image deblurring over Restormer @Zamir2021Restormer. We also provide the visual results as shown in Fig. \[fig:hide\]. Our method can combine the advantages of the different fused methods, which leads to the best results.

### Image deraining.

We also select seven image deraining models, *i.e.* DerainNet @fu2017clearing, SPAIR @purohit2021spatially, PreNet @ren2019progressive, MSPFN @Kui_2020_CVPR, MPRNet @mehri2021mprnet, HINet @chen2021hinet, and Restormer @Zamir2021Restormer, for comparison. And PreNet @ren2019progressive, MSPFN @Kui_2020_CVPR, MPRNet @mehri2021mprnet, HINet @chen2021hinet, and Restormer @Zamir2021Restormer is taken as the deraining pool. As listed in Tab. \[tab:deraining\], our BDE can outperform the state-of-the-art method, *i.e.* Restormer, $>0.3$dB. Fig. \[fig:rain100l\_16\] is visual results of different comparison methods. It indicates that our method can well fuse different datasets with a variety of rain streaks.

### Single image super-resolution

We select five state-of-the-art SISR methods, *i.e.* RCAN @zhang2020residual, SAN @dai2019second, RRDB @wang2018esrgan, IPT @chen2021pre, and SwinIR @liang2021swinir, as the SR pool. And EDSR @lim2017enhanced and RNAN @zhang2019rnan are also selected for comparison. As listed in Tab. \[tab:sisr\], compared to the state-of-the-art SwinIR model, the PSNR can obtain $\sim$0.1dB gain. Hence, our proposed method can be applied to more low-level tasks. Employing a similar strategy for the task would give a more tangible method.

Ablation study
--------------

We design two experiments to notarize the role of different parts in our proposed method: i) why and how to estimate scoring map; ii) influence on the type and number of fused denoisers.

<span>**<span>Different approaches to fuse denoisers.</span>**</span> As mentioned in Sec. \[ensemble\_model\], it exists several approaches for fusing different models in image restoration tasks. We implement those approaches, *i.e.* @liao2015video [@liu2017robust; @cho2021compression; @yang2020image], by ourselves due to no public code or different tasks. @liu2020improved is not reproduced due to only fusing two denoisers and lack of implementation details to fuse more denoisers. The same denoiser pool with our BDE is utilized for a fair comparison. Note that @yang2020image stacks the denoisers in sequence. As listed in Tab. \[tab:DND\_c\], our BDE can easily outperform the other fusion strategies which verify the effectiveness of the proposed method. The reason why they are far inferior to our method is that they focused on fusing the models under the same condition, *e.g.* same noise type, noise level, and even same architecture. Moreover, they all take fixed methods as input, which limits their application to real scenarios.

\[tab:DND\_c\]

<span>**<span>Effect of Bayesian deep learning.</span>**</span> We examine the necessity of Bayesian deep learning (BDL) in our BDE by training three variants: 1) BDE$_{Normal}$ is a normal U-Net architecture without MC dropout layers and estimating uncertainty map $\boldsymbol{\sigma}$. 2) BDE(-MC) removes MC dropout layers. 3) BDE$\left(-\boldsymbol{\sigma}\right)$ doesn't estimate uncertainty map $\boldsymbol{\sigma}$. One can find the generalization ability of the model becomes weaker without BDL. Both MC dropout layers and estimating uncertainty map $\boldsymbol{\sigma}$ improve the denoising performance.

<span>**<span>The effectiveness of estimating scoring map.</span>**</span> We utilize three variants of the scoring network to verify the effectiveness of estimating the scoring map: 1) BDE($_{PaW}$) denotes patch-level prediction instead of pixel-wise prediction; 2) BDE$_{Weight}$ denotes directly estimating the weighting map without the SoftMax function. Table. \[tab:vardnd\] lists the results of different variants, and we can obtain the following conclusions: i) Instead of predicting a single score for one patch, the pixel-wise prediction can be more effective to combine different denoisers' results. ii) Because of the SoftMax function, our prediction can be more robust.

<span>**<span>The role of the different loss functions.</span>**</span> In Eq.(\[eq:weights\]), our loss function contain two terms, *i.e.* negative log-likelihood loss in Eq.(\[eq:nlltv\]) and fusing loss in Eq.(\[eq:sum\]). We remove one of them and denote them as BDE(-${\ell_{nll}}$) and BDE(-${\ell_{fuse}}$) respectively. As shown in Table. \[tab:vardnd\], the performance of the proposed method is obtained by their joint effort.

<span>**<span>The number of fused denoisers.</span>**</span> Fig. \[fig:num\_denoisers\] shows the influence of the fused denoisers' number. The red line denotes that only fuses the real-world denoisers. The base denoiser is Uformer @wang2021uformer, and then NBNet @cheng2021nbnet, MIRNet @zamir2020learning, MPRNet @mehri2021mprnet, and HINet are added in turn. Both the green line and the blue line that fuse both real-world and synthetic denoisers. The base denoiser of the green line is Uformer @wang2021uformer, and then CBDNet @guo2019toward, NBNet @cheng2021nbnet, MPRNet @mehri2021mprnet, HINet @cheng2021nbnet, and GMSNet @song2020grouped are added in turn. The base denoiser of the blue line is GMSNet @song2020grouped, and then CBDNet @guo2019toward, NBNet @cheng2021nbnet, MIRNet @zamir2020learning, MPRNet @mehri2021mprnet, and HINet @cheng2021nbnet are added in turn. One can find that the PSNR performance can get a continuous rise with the number of denoisers increasing, but the growth rate slows down after fusing 3 denoisers. Compared with the red and green lines, the PSNR performance can obtain a remarkable gain after combining the denoisers with different training datasets. Compared with the red and blue lines, the increase rate of the red line is lower due to all denoisers overfitting on the SIDD dataset. So we suggest that the fused denoisers should own different characteristics, especially training on different datasets.

![The influence of the fused denoisers' number. The red line denotes that only fuses the real-world denoisers, and the base denoiser is Uformer. While results of both the green line and the blue line are that fuse both real-world and synthetic denoisers. The base denoiser of the green line is Uformer, and the base denoiser of the blue line is GMSNet.<span data-label="fig:num_denoisers"></span>](num_denoisers.pdf){width="49.00000%"}

We provide the results of fusing four of five different denoisers in Tab. \[tab:number\]. Then we can get the following conclusions: i) fusing denoisers with different training datasets can achieve better generalization than one dataset; ii) greater difference of network, stronger complementary; iii) collecting denoisers with different training datasets and architectures rather than fusing more denoisers.

\[tab:number\]

<span>**<span>Complementarity of the different denoisers' results</span>**</span> The complementarity of the different denoisers' results comes from two folds: i) The training datasets. Assume that two denoisers with the same architecture are trained on two extremely different datasets, *e.g.* real noisy dataset and synthetic dataset, respectively. As shown in Fig. \[fig:num\_denoisers\], only fusing denoisers that only train on SIDD cannot achieve big progress on the DND dataset. While the performance can achieve big gain by fusing denoisers training on different datasets. ii) The architecture. For example, Uformer @wang2021uformer considers more global information, while GMSNet @song2020grouped pays more attention to local information.

Limitations
-----------

The limitations of our proposed method can be summarized in two aspects: On the one hand, the fused denoisers should have different characteristics to form complementarity. Then the denoising performance can be further improved. If not so, the performance is limited, such as results on SIDD. On the other hand, our method is time-consuming due to inference time on different denoisers and estimating weighting maps for each denoised result, even though it only needs 0.002 seconds to estimate a scoring map with size $256\times256$ . This limits our method of applying to real-time scenes. However, for real-time scenarios, BDE has been trained to generalize well compared to single small models, and therefore we leverage it as an oracle to get noise-free results from the unpaired inputs, and subsequently, any practical small model designed for real-time inference can be trained in a supervised manner.

Conclusion {#secV}
==========

In this paper, we propose an effective Bayesian deep ensemble (BDE) method to exploit different external prior, *i.e.* different well-trained denoisers, for real-world image denoising and other image restoration tasks. Our BDE takes one of the noise removal results with its noisy image as input and then predicts the scoring map. After obtaining all scoring maps, we use the SoftMax function to obtain the probability maps to combine the images in a pixel-wise approach. Without the corresponding training dataset, our BDE method can fuse different denoisers to achieve state-of-the-art performance on DND dataset, which indicates the robustness and generalization of our proposed ensemble method. Moreover, by combining five Gaussian denoisers, it can obtain better denoising performance than CBDNet. Hence, for an unknown type of noisy image, our BDE method is a potentially effective approach to provide a clean result. For other image restoration tasks, our BDE method can be directly applied and also achieve an amazing effect. However, we also point out the limitations of our work, such as the special demands of different denoisers and time-consuming. In further work, we will improve those limitations and seek an unsupervised approach for ensemble learning.

[^1]: P. Liu, H. Zhang, D. Ren, and W. Zuo are with the School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China. e-mail: zwatts@example.org; zwatts@example.org; zwatts@example.org; zwatts@example.org.

[^2]: J. Wang and Y. Wang are with Megvii Inc, Beijing, China. e-mail: zwatts@example.org; zwatts@example.org.

{'source': 'arXiv_out/2206'}