[Re]
Spatial-Adaptive Network for Single Image Denoising
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility
Summary
In this study we present our results and experience during replicating the paper titled "Spatial-Adaptive Network for
Single Image Denoising".
This paper proposes novel spatial-adaptive denoising architecture for efﬁcient noise removalby leveraging the deformable convolutions to adapt spatial information (i e. edges and textures).
We have implementedthe model from scratch in PyTorch framework and then have conducted real and synthetic noise experiments on thecorresponding datasets.
We have achieved to reproduce the results qualitatively and quantitatively.
Scope of Reproducibility
The original paper proposes an encoder-decoder structure exploiting a residual spatial-adaptive block and a contextblock to capture multi-scale information for achieving the state-of-the-art on real and synthetic noise removal
Methodology
We have implemented the model, namely SADNet , from scratch in PyTorch as described in the paper, and also adoptedthe training loop and proposed blocks from the author’s code.
Since the weight initialization of proposed blocks wasnot implicitly deﬁned in the paper, we have decided to use the default initialization method for convolutional layers in
PyTorch ( i.e. Kaiming).
Experiments have been completed on a single RTX 2080 Ti in 3 days for each, and it requires∼ 3 GB GPU memory for training, and ∼ 8 GB CPU memory for loading the data, due to the ﬁle structure of datasets.
Results
We have achieved to reproduce the results qualitatively and quantitatively on synthetic and noise removal tasks.
SADNethas the capacity to learn to remove the synthetic and real noise in images, and it produces visually-plausible outputseven after a few epochs.
Moreover, we have employed SSIM and PSNR metrics to measure the quantitative performancefor all settings.
The quantitative results on both tasks are on-par when compared to the reported results in the paper.
What was easy
The code was open-source, and implemented in PyTorch, hence adopting the training loop and proposed blocks to our implementation facilitated our reproduction study.
The loss function is straightforward and the architecture has a
U-Net-like structure, so that we could achieve to implement the architecture in a fair time
What was difﬁcult
Due to the lack of compatibility with the current versions of PyTorch and TorchVision and the dependency on an external implementation of deformable convolutions we have encountered several issues during our implementation
Then we have considered to re-implement residual spatial-adaptive block and context block from scratch for deferring these dependencies however we could not achieve it just by referring to the paper in limited time.
Therefore we have decided to directly use the provided blocks the
Communication with original authors
We did make any contact with the authors since we achieved to solve the issues encountered during the implementa- tion of SADNet by examining the author’s
Introduction
Recent works [2, 3, 4, 5] have shown that the previous assumption of an identically-distributed additive white Gaussiannoise (AWGN) is not an accurate representation of the real noise occurring in images.
Traditional denoiser architectureslack the ability to adapt textures and edges, and thus miss the details while denoising, due to the over-smoothingbehaviour of CNNs.
A workaround to this problem is implementing a deeper network, however, such a practiceintroduces a more complex model with its computational burden.
In the original paper [1], an encoder-decoder architecture consisting a residual spatial-adaptive block, namely RSAB, isproposed for removing spatially-variant and channel-dependent noise while processing larger regions in each step byutilizing deformable convolutions.
As the main claim of the paper, this method produces better performance than thecompared methods in given benchmark, and also for the synthetic noise removal task.
In this reproducibility report, we studied SADNet architecture for both real and synthetic noise removal in detail, whichcontains implementing the architecture described in the paper, running the experiments, reporting the important detailsabout certain issues encountered during reproducing, and comparing the obtained results with the ones reported in theoriginal paper.
Scope of reproducibility
The main idea of the paper is to present a spatial-adaptive architecture with encoder-decoder structure which capturesthe relevant features from the complex image content while removing real noise appearing in images.
Residualspatial-adaptive block (RSAB) makes it possible to achieve this in an efﬁcient manner.
The proposed model, namely SADNet, claims to outperform the state-of-the-art performances in SSIM and PSNRmetrics with a moderate run-time.
To validate these claims, we try to investigate the following questions:
• Is the implementation details described in the paper and provided code sufﬁcient for replicating the quantitative
results reported in the paper?
• Are the qualitative results visually-plausible?
• Are the replicated quantitative results competitive enough?
• Could our replication obtain a proximate denoising duration compared to the reported results in the original
paper
Methodology
We have implemented the model, namely SADNet, from scratch in PyTorch [6], as described in the paper by adopting
RSAB, Context block and Offset block from the author’s code.
The implementation of residual blocks (ResBlock) in theauthor’s code differs from the common residual block implementation [17] by not using the output activation.
In contrastto the common practice of applying a nonlinear activation function to the output, their ResBlock implementation directlyforwards its output to the next level layers.
At this point, the authors handle those activations at the model scope.
Wealso removed Batch Normalization [7] from the residual blocks as proposed by the original paper.
To enhance thereadability of the model structure in our implementation, we imported those activation functions back in to ResBlock.
The deformable convolutions in RSAB are implemented in CUDA, hence we used NVIDIA GPUs with the relevant
CUDA driver.
For validating the reported results on real noisy images, we have implemented the data loaders, which are missing inthe author’s code.
Furthermore, we integrated WandB [8] library to the training loop in order to track our experimentsduring training.
3.1 SADNet
SADNet is an encoder-decoder architecture with skip connections which favors spatial adaptability and large receptiveﬁeld over deeper networks for the well-studied denoising task.
The proposed model aims to achieve the state-of-the-artdenoising performance while maintaining the computational complexity by exploiting residual spatial-adaptive block(RSAB), Context Block and Offset Block.
The visual representation of SADNet architecture is shown in Figure 1, andalso the structural details about our implementation of SADNet can be seen in Table 1.
Regions with the sharp texture changes in an image, typically edges and corners, raise difﬁculties for training the regularconvolutions, due to its ﬁxed-size weighting mechanism.
Such regions, where different textures co-occur in a particularreceptive ﬁeld of the regular convolutions, are simply ignored during the weighting process, due to the ﬁxed size kernels.
To address this issue, self-similarity weighting is attained via modulated deformable convolutions [25] in RSAB.
Thekernels of the deformable convolutions have a learnable offset for each location in an image, and thus it has the capacityto adapt to the spatial texture changes.
The formula of modulated deformable convolutions can be seen as follows
[Equation]
where ∆ p i denotes the learnable offset for location p i , and ∆ m i is the extra degree of freedom for adjusting themodulation scalar between [0 , 1] .
The nature of the decoder architectures enforces to transform the feature maps from coarse to ﬁne at each scale.
Forlearning the offsets more accurately in RSABs, the offsets ∆ p s − 1 and the modulation scalars ∆ m s − 1 from the previousscale are further transferred into the current scale s with the help of Offset Blocks.
The offset transfer is formulated as
[Equation]
) where F up denotes the up-sampling operation.
RSAB receive the extracted features and the reconstructed featuresfrom the previous scale conveyed by the Offset Block.
The inputs are then fused through a modulated deformableconvolution layer with a subsequent regular convolution layer.
Moreover, a skip connection similar to ResBlock isemployed to enhance the information transferring.
At this point, RSAB can be formulated as,
[Equation]
where F cn and F dcn denote regular convolution and modular deformable convolution, respectively.
Lastly, F act standsfor the leaky ReLU activation function [15] with a negative slope of 0 . 2 .
Another introduced block is the Context Block, which resides at the bottleneck of the model.
To increase the size ofthe receptive ﬁelds while preserving the spatial resolution, Context Block is employed for the model, just betweenthe encoder and decoder structures.
Furthermore, unlike the common implementation of Context Block, Batch
Normalization layer is removed in the published code, and only four dilation rates are used, which are 1, 2, 3 and 4.
Following the original paper, we used L 1 loss for training our model on real-noise image datasets, and L 2 loss fortraining on synthetic image datasets.
[Table]
The original paper uses SIDD Medium [9] and RENOIR [10] datasets during training for denoising real noisy imagesand reports the qualitative and quantitative results on DND [11] and SIDD validation [9] datasets.
Since the author’scode only provides the data loader for synthetic image datasets, we have integrated our SIDD validation data loaderimplementation and the DND test script provided by TU Darmstadt [11] to our pipeline.
For synthetic noise removal,additive white Gaussian noise with standard deviation of 30, 50 and 70 have been added to DIV2K dataset [12], whichconsists 800 high resolution images.
To validate the performance of SADNet on synthetic noise removal task, themodels are tested on BSD68 [13] and Kodak24 [14] datasets processed with the same noise addition mechanism.
Both test and validation data for all settings are composed of high resolution images.
Therefore, they are fed to themodel as 128x128 patches cropped by ﬁxed coordinates, as described in the paper.
We have applied 90 ◦ rotation,horizontal and vertical ﬂipping to the images during training, following the practice in the paper.
3.3 Hyperparameters
In our replication study, we used the ADAM optimizer [16] with β 1 = 0 .
9 , β 2 = 0 .
999 , and (cid:15) =
1 e − 8 , with an initiallearning rate of 1 e − 4 during training, as described in the paper.
The provided code initializes the weights of theconvolutional layers in all blocks with Xavier Uniform method [23].
Since this choice has not been discussed in thepaper, we left each convolution layer initialized by the default weight initialization method in PyTorch ( i.e. Kaiming[24]) in our experiments.
3.4 Experimental setup
In this study, we have followed the same training procedures for all setting, and employed SSIM and PSNR values asperformance metrics, as described in the original paper.
The parameters for all training settings can be found in the conﬁguration ﬁle in our GitHub repository.
Our implementation and the trained weights are open-sourced, and can beaccessed at https://anonymous.4open.science/r/96f7ac9a-be7b-4d7a-ac10-44da93e6948c/ .
3.5 Computational requirements
The experiments have been conducted on a single RTX 2080Ti for approximately 3 days, and only requires high GPUmemory, mostly due to modulated deformable convolutions.
It requires ∼ 3 GB GPU memory for training, and ∼ 8 GB
CPU memory for loading the data, due to the ﬁle structure of datasets.
[Table]
[Figure] [Figure]
Results
We have implemented the model from scratch by following the descriptions presented in the original paper, and thenachieved to replicate the claimed results by referring to the published code.
Overall, our implementation of SADNetachieved on-par performances in SSIM and PSNR metrics on test datasets, and we also validated the results on bothdenoising tasks by examining their qualitative results.
As shown in Table 2, our quantitative results on SIDD sRGB validation dataset has 39.41 PSNR value, which is only0.12% less than the one reported in the original paper.
Moreover, the average duration of a single inference of SADNetis 26.7 ms.
according to the paper, while our implementation of SADNet completes the single inference on 25.9 ms.
The visual comparisons of real noise removal of the images are shown in Figure 2 and Figure 3.
The samples are from
SIDD validation dataset, according to the ones reported in the paper.
The ﬁrst two rows in these ﬁgures are directlytaken from the original paper for further comparison with our replication results, which can be seen in the third row.
Onthe results from the original paper, SADNet mainly differs from the compared methods by generating a distinct clearcontinuous stripe texture on the background while preserving the object surface appearance.
Our replication clearlyshares the similar behaviour.
Therefore, we can state that the replicated quantitative results on real noise removal taskare competitive enough, and also supports the main claim in the original paper.
[Figure]
Similarly, Table 3 demonstrates that the results of our SADNet implementation achieves on-par PSNR values withthe ones reported in the paper for different noise levels ( i.e. σ ∈ { 30 , 50 , 70 } ) on BSD68 and Kodak24 datasets.
Particularly, we have obtained better results than all other compared methods and the reported SADNet results on
Kodak24 dataset for all noise levels.
Moreover, the replicated SADNet model imitates the qualitative results of theoriginal model on both datasets.
Although the images from Kodak24 and BSD68 are heavily exposed to the syntheticnoise, SADNet has the ability to remove noise, and to generate well-deﬁned textures when compared to the recentworks.
As shown in Figure 4, all other compared methods have smoothed the texture and swept away the feather details,meanwhile the original implementation of SADNet and ours achieve to generate more plausible feather-like texture.
Similar to the previous example, in Figure 5, the clothing details are signiﬁcantly preserved, especially pilling on thetop-left part of the cloth and the vertical texture details on the cloth.
The ground truth of DND validation set is private, and thus it is not possible to locally validate the results on this dataset.
Despite of several attempts to submit our results to DND online validation system, we could not obtain SSIM and PSNRresults, due to the server error.
We have tried to contact with DND Team, but we could not get any advice for solvingthis issue.
[Table]
DnCNN [19] 31.
MemNet [20] 29.
FFDNet [21] 31.
RNAN [22] 31.
RIDNet [5] 31.
SADNet [1] 31.
SADNet (ours) 32.
[Figure]
Discussion
The qualitative results generated with our replication strongly resemble to the presented results, and differs from theother compared studies.
According to these results, we can state that our implementation of SADNet consistently yieldsvisually-plausible results on both real and synthetic noisy images, and supports the claims of the original paper.
Inaddition, our experiments ﬁrmly correlates with the reported PSNR values.
Overall, the paper and the provided code was sufﬁcient for replicating the results on real and synthetic noise removal.
For re-implementing the model from scratch, we have only referred to the paper, and ended up on-par performance withthe ones in the paper on all settings.
Lastly, to provide an insight for run-time on different hardware, our replication has 25.9 ms. inference run-time on realnoise removal task, whereas the reported run-time duration is 26.7 ms.
Note that we used a single RTX 2080Ti GPUduring our experiments, while a single GTX 1080Ti GPU is used in the original study, and we assume that this is thereason of this difference.
5.1 What was easy
The code was open-source, and implemented in PyTorch, hence adopting the training loop and proposed blocks toour implementation facilitated our reproduction study.
The loss function is straightforward and the architecture has a
U-Net-like structure, so that we could achieve to implement the architecture in a fair time.
5.2 What was difﬁcult
Due to the lack of compatibility with the current versions of PyTorch and TorchVision and the dependency on an external
CUDA implementation of deformable convolutions, we have encountered several issues during our implementation.
Then, we have considered to re-implement residual spatial-adaptive block and context block from scratch for deferringthese dependencies, however, we could not achieve this just by referring to the paper.
Therefore, we have decided todirectly use the provided blocks as in the author’s code.
5.3 Communication with original authors
We did not make any contact with the authors since we achieved to solve the issues encountered during the implementation of SADNet by examining the author’s code.