R ESIDUAL N ON - LOCAL A TTENTION N ETWORKS FOR I MAGE R ESTORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a residual non-local attention network for high-quality image restoration.
Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features.
To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts.
Speciﬁcally, we design trunk branch and (non-)local mask branch in each (non-)local attention block.
The trunk branch is used to extract hierarchical features.
Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions.
The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map.
Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network.
Our proposed method can be generalized for various image restoration tasks, including image denoising, demosaicing, compression artifacts reduction, and super-resolution.
Experiments show that our method achieves comparable or better results compared with recently leading methods quantitatively and visually.
I NTRODUCTION
Image restoration aims to recover high-quality (HQ) images from their corrupted low-quality (LQ) observations and plays a fundamental role in various high-level vision tasks.
It is a typical ill-posed problem due to the irreversible nature of the image degradation process.
Some most widely studied image restoration tasks include image denoising, demosaicing, and compression artifacts reduction.
By distinctively modelling the restoration process from LQ observations to HQ objectives, i.e., without assumption for a speciﬁc restoration task when modelling, these tasks can be uniformly addressed in the same framework.
Recently, deep convolutional neural network (CNN) has shown extraordinary capability of modelling various vision problems, ranging from low-level (e.g., image denoising (Zhang et al., 2017a), compression artifacts reduction (Dong et al., 2015), and image super-resolution
(Kim et al., 2016; Lai et al., 2017; Tai et al., 2017; Lim et al., 2017; Zhang et al., 2018a;
Haris et al., 2018; Wang et al., 2018c; Zhang et al., 2018b; Wang et al., 2018b)) to high-level (e.g., image recognition (He et al., 2016)) vision applications.
Stacked denoising auto-encoder (Vincent et al., 2008) is one of the best well-known CNN based model for image restoration.
Dong et al.
proposed SRCNN (Dong et al., 2014) for image superresolution and ARCNN (Dong et al., 2015) for image compression artifacts reduction.
Both SRCNN and ARCNN achieved superior performance against previous works.
By introducing residual learning to ease the training difﬁculty for deeper network, Zhang et al.
proposed DnCNN (Zhang et al., 2017a) for image denoising and compression artifacts reduction.
The denoiser prior was lately introduced in IRCNN (Zhang et al., 2017b) for fast image restoration.
Mao et al. proposed a very deep fully convolutional encoding-decoding framework with symmetric skip connections for image restoration (Mao et al., 2016).
Tai et al.
later proposed a very deep end-to-end persistent memory network (MemNet) for image restoration (Tai et al., 2017) and achieved promising results.
These
CNN based methods have demonstrated the great ability of CNN for image restoration tasks.
However, there are mainly three issues in the existing CNN based methods above.
First , the receptive ﬁeld size of these networks is relatively small.
Most of them extract features in a local way with convolutional operation, which fails to capture the long-range dependencies between pixels in the whole image.
A larger receptive ﬁeld size allows to make better use of training inputs and more context information.
This would be very helpful to capture the latent degradation model of LQ images, especially when the images suffer from heavy corruptions.
Second , distinctive ability of these networks is also limited.
Let’s take image denoising as an example.
For a noisy image, the noise may appear in both the plain and textural regions.
Noise removal would be easier in the plain area than that in the textural one.
It is desired to make the denoising model focus on textual area more.
However, most previous denoising methods neglect to consider different contents in the noisy input and treat them equally.
This would result in over-smoothed outputs and some textural details would also fail to be recovered.
Third , all channel-wise features are treated equally in those networks.
This naive treatment lacks ﬂexibility in dealing with different types of information (e.g., low- and high-frequency information).
For a set of features, some contain more information related to HQ image and the others may contain more information related to corruptions.
The interdependencies among channels should be considered for more accurate image restoration.
To address the above issues, we propose the very deep residual non-local attention networks (RNAN) for high-quality image restoration.
We design residual local and non-local attention blocks as the basic building modules for the very deep network.
Each attention block consists of trunk and mask branches.
We introduce residual block (He et al., 2016; Lim et al., 2017) for trunk branch and extract hierarchical features.
For mask branch, we conduct feature downscaling and upscaling with largestride convolution and deconvolution to enlarge receptive ﬁeld size.
Furthermore, we incorporate non-local block in the mask branch to obtain residual non-local mixed attention.
We apply RNAN for various restoration tasks, including image denoising, demosaicing, and compression artifacts reduction.
Extensive experiments show that our proposed RNAN achieves state-of-the-art results compared with other recent leading methods in all tasks.
To the best of our knowledge, this is the ﬁrst time to consider residual non-local attention for image restoration problems.
The main contributions of this work are three-fold:
• We propose the very deep residual non-local networks for high-quality image restoration.
The powerful networks are based on our proposed residual local and non-local attention blocks, which consist of trunk and mask branches.
The network obtains non-local mixed attention with non-local block in the mask branch.
Such attention mechanis helps to learn local and non-local information from the hierarchical features.
• We propose residual non-local attention learning to train very deep networks by preserving more low-level features, being more suitable for image restoration.
Using non-local lowlevel and high-level attention from the very deep network, we can pursue better network representational ability and ﬁnally obtain high-quality image restoration results.
• We demonstrate with extensive experiments that our RNAN is powerful for various image restoration tasks.
RNAN achieves superior results over leading methods for image denoising, demosaicing, compression artifacts reduction, and super-resolution.
In addition, RNAN achieves superior performance with moderate model size and performs very fast.
R ELATED W ORK
Non-local prior.
As a classical ﬁltering algorithm, non-local means (Buades et al., 2005) is computed as weighted mean of all pixels of an image.
Such operation allows distant pixels to contribute to the response of a position at a time.
It was lately introduced in BM3D (Dabov et al., 2007b) for image denoising.
Recently, Wang et al.
(2018a) proposed non-local neural network by incorporating non-local operations in deep neural network for video classiﬁcation.
We can see that those methods mainly introduce non-local information in the trunk pipeline.
Liu et al.
(2018) proposed non-local recurrent network for image restoration.
However, in this paper, we mainly focus on learning nonlocal attention to better guide feature extraction in trunk branch.
Attention mechanisms.
Generally, attention can be viewed as a guidance to bias the allocation of available processing resources towards the most informative components of an input (Hu et al.,
Recently, tentative works have been proposed to apply attention into deep neural networks (Wang et al., 2017; Hu et al., 2017).
It’s usually combined with a gating function (e.g., sigmoid) to rescale the feature maps.
Wang et al.
(2017) proposed residual attention network for image classiﬁcation with a trunk-and-mask attention mechanism.
Hu et al.
(2017) proposed squeezeand-excitation (SE) block to model channel-wise relationships to obtain signiﬁcant performance improvement for image classiﬁcation.
In all, these works mainly aim to guide the network pay more attention to the regions of interested.
However, few works have been proposed to investigate the effect of attention for image restoration tasks.
Here, we want to enhances the network with distinguished power for noise and image content.
Image restoration architectures.
Stacked denoising auto-encoder (Vincent et al., 2008) is one of the most well-known CNN-based image restoration method.
(Dong et al., 2015) proposed ARCNN for image compression reduction with several stacked convolutional layers.
With the help of residual learning and batch normalization (Ioffe & Szegedy, 2015), Zhang et al.
proposed
DnCNN (Zhang et al., 2017a) for accurate image restoration and denoiser priors for image restoration in IRCNN (Zhang et al., 2017b).
Recently, great progresses have been made in image restoration community, where Timofte et al.
(Timofte et al., 2017), Ancuti et al.
(Ancuti et al., 2018), and Blau et al.
(Blau et al., 2018) lead the main competitions recently and achieved new research status and records.
For example, Wang et al.
(Wang et al., 2018c) proposed a fully progressive image SR approach.
However, most methods are plain networks and neglect to use non-local information.
[Figure]
for image restoration.
‘Conv’, ‘RNAB’, and ‘RAB’ denote convolutional layer, residual non-local attention block, and residual local attention block respectively.
Here, we take image denoising as a task of interest.
R ESIDUAL N ON - LOCAL A TTENTION N ETWORK FOR I MAGE R ESTORATION 3. F RAMEWORK
The framework of our proposed residual non-local attention network (RNAN) is shown in Figure 1.
Let’s denote I L and I H as the low-quality (e.g., noisy, blurred, or compressed images) and highquality images.
The reconstructed image I R can be obtained by
[Equation]
where H RNAN denotes the function of our proposed RNAN.
With the usage of global residual learning in pixel space, the main part of our network can concentrate on learning the degradation components (e.g., noisy, blurring, or compressed artifacts).
The ﬁrst and last convolutional layers are shallow feature extractor and reconstruction layer respectively.
We propose residual local and non-local attention blocks to extract hierarchical attentionaware features.
In addition to making the main network learn degradation components, we further concentrate on more challenging areas by using local and non-local attention.
We only incorporate residual non-local attention block in low-level and high-level feature space.
This is mainly because a few non-local modules can well offer non-local ability to the network for image restoration.
Then RNAN is optimized with loss function.
Several loss functions have been investigated, such as L 2 (Mao et al., 2016;
Zhang et al., 2017a;
Tai et al., 2017; Zhang et al., 2017b), L 1 (Lim et al., 2017; Zhang et al., 2018c), perceptual and adversarial losses (Ledig et al., 2017).
To show the effectiveness of our RNAN, we choose to optimize the same loss function (e.g., L 2 loss function) as previous works.
Given a training set (cid:8) I iL , I iH (cid:9) N i =1 , which contains N low-quality inputs and their high-quality counterparts.
The goal of training RNAN is to minimize the L 2 loss function
[Equation]
where (cid:107)·(cid:107) 2 denotes l 2 norm.
As detailed in Section 4, we use the same loss function as that in other compared methods.
Such choice makes it clearer and more fair to see the effectiveness of our proposed RNAN.
Then we give more details to residual local and non-local attention blocks.
RB
RB
RB q q
RB
RB t
C onv R e L
U
C onv
RB
RB m
RB
RB 2m
RB
RB m
S ub s ca l e
U p s ca l e
C onv
S i g m o i d
[Figure]
3. R ESIDUAL N ON - LOCAL A TTENTION B LOCK
Our residual non-local attention network is constructed by stacking several residual local and nonlocal attention blocks shown in Figure 2.
Each attention block is divided into two parts: q residual blocks (RBs) in the beginning and end of attention block.
Two branches in the middle part: trunk branch and mask branch.
For non-local attention block, we incorporate non-local block (NLB) in the mask branch, resulting non-local attention.
Then we give more details to those components.
3.2. T RUNK B RANCH
As shown in Figure 2, the trunk branch includes t residual blocks (RBs).
Different from the original residual block in ResNet (He et al., 2016), we adopt the simpliﬁed RB from (Lim et al., 2017).
The simpliﬁed RB (labelled with blue dashed) only consists of two convolutional layers and one
ReLU (Nair & Hinton, 2010), omitting unnecessary components, such as maxpooling and batch normalization (Ioffe & Szegedy, 2015) layers.
We ﬁnd that such simpliﬁed RB not only contributes to image super-resolution (Lim et al., 2017), but also helps to construct very deep network for other image restoration tasks.
Feature maps from trunk branch of different depths serve as hierarchical features.
If attention mechanism is not considered, the proposed network would become a simpliﬁed ResNet.
With mask branch, we can take channel and spatial attention to adaptively rescale hierarchical features.
Then we give more details about local and non-local attention.
3.2. M ASK B RANCH
As labelled with red dashed in Figure 2, the mask branches used in our network include local and non-local ones.
Here, we mainly focus on local mask branch, which can become a non-local one by using non-local block (NLB, labelled with green dashed arrow).
The key point in mask branch is how to grasp information of larger scope, namely larger receptive ﬁeld size, so that it’s possible to obtain more sophisticated attention map.
One possible solution is to perform maxpooling several times, as used in (Wang et al., 2017) for image classiﬁcation.
However, more pixel-level accurate results are desired in image restoration.
Maxpooling would lose lots of details of the image, resulting in bad performance.
To alleviate such drawbacks, we choose to use large-stride convolution and deconvolution to enlarge receptive ﬁeld size.
Another way is considering non-local information across the whole inputs, which will be discussed in the next subsection.
From the input, large-stride (stride ≥ 2) convolutional layer increases the receptive ﬁeld size after m
RBs.
After additional 2 m RBs, the downscaled feature maps are then expanded by a deconvolutional u :×v :×g :×
S o f t m a x z :×
H × W × 2C
H × W × C
H × W × C H × W × C
HW × C HW × C
C × HW
HW × HW
HW × C
H
× W × 2C H
×
W × 2 C
H × W ×
C layer ( as transposed convolutional layer).
The upscaled features are further forwarded through m RBs and one 1 × 1 convolutional layer.
Then a sigmoid layer normalizes the output values, ranging in [0 , 1] .
Although the receptive ﬁeld size of the mask branch is much larger than that of the trunk branch, it cannot cover the whole features at a time.
This can be achieved by using non-local block (NLB), resulting in non-local mixed attention.
3.2. N ON - LOCAL M IXED A TTENTION
As discussed above, convolution operation processes one local neighbourhood at a time.
In order to obtain better attention maps, here we seek to take all the positions into consideration at a time.
Inspired by classical non-local means method (Buades et al., 2005) and non-local neural networks (Wang et al., 2018a), we incorporate non-local block (NLB) into the mask branch to obtain non-local mixed attention (shown in Figure 3).
The non-local operation can be deﬁned as
[Equation]
where i is the output feature position index and j is the index that enumerates all possible positions.
x and y are the input and output of non-local operation.
The pairwise function f ( x i , x j ) computes relationship between x i and x j .
The function g ( x j ) computes a representation of the input at the position j .
As shown in Figure 3, we use embedded Gaussian function to evaluate the pairwise relationship
[Equation]
where W u and W v are weight matrices.
As investigated in (Wang et al., 2018a), there are several versions of f , such as Gaussian function, dot product similarity, and feature concatenation.
We also consider a linear embedding for g : g ( x j ) =
W g x j with weight matrix W g .
Then the output z at position i of non-local block (NLB) is calculated as
[Equation]
where W z is a weight matrix.
For a given i , (cid:80) ∀ j f ( x i , x j ) / (cid:80) ∀
j f ( x i , x j ) in Eq.
3 becomes the softmax computation along dimension j .
The residual connection allows us to insert the NLB into pretrained networks (Wang et al., 2018a) by initializing W z as zero.
With non-local and local attention computation, feature maps in the mask branch are ﬁnally mapped by sigmoid function
[Equation] [Equation]
where i ranges over spatial positions and c ranges over feature channel positions.
Such simple sigmoid operation is applied to each channel and spatial position, resulting mixed attention (Wang et al., 2017).
As a result, the mask branch with non-local block can produce non-local mixed attention.
However, simple multiplication between features from trunk and mask branches is not powerful enough or proper to form very deep trainable network.
We propose residual non-local attention learning to solve those problems.
3. R ESIDUAL N ON - LOCAL A TTENTION L EARNING
How to train deep image restoration network with non-local mixed attention remains unclear. Here we only consider the trunk and mask branches, and residual connection with them (Figure 2).
We focus on obtaining non-local attention information from the input feature x .
It should be noted that one form of attention residual learning was proposed in Wang et al.
(2017), whose formulation is
[Equation]
RNA
[Equation]
We ﬁnd that this form of attention learning is not suitable for image restoration tasks.
This is mainly because Eq.
7 is more suitable for high-level vision tasks (e.g., image classiﬁcation), where low-level features are not preserved too much.
However, low-level features are more important for image restoration.
As a result, we propose a simple yet more suitable residual attention learning method by introducing input feature x directly.
We compute its output H RNA ( x ) as
[Equation]
where H trunk ( x ) and H mask ( x ) denote the functions of trunk and mask branches respectively.
Such residual learning tends to preserve more low-level features and allows us to form very deep networks for high-quality image restoration tasks with stronger representation ability.
3. I MPLEMENTATION D ETAILS
Now, we specify the implementation details of our proposed RNAN.
We use 10 residual local and non-local attention blocks (2 non-local one).
In each residual (non-)local block, we set q, t, m = 2 , 2 , 1 .
We set 3 × 3 as the size of all convolutional layers except for those in non-local block and convolutional layer before sigmoid function, where the kernel size is 1 × 1.
Features in RBs haveﬁlters, except for that in the non-local block (see Figure 3), where C = 32 .
In each training batch, 16 low-quality (LQ) patches with the size of 48 × 48 are extracted as inputs.
Our model is trained by ADAM optimizer (Kingma & Ba, 2014) with β 1 = 0 .
9 , β 2 = 0 .
999 , and (cid:15) =
10 − 8 .
The initial learning rate is set to 10 − 4 and then decreases to half every 2 × 10 5 iterations of back-propagation.
We use PyTorch (Paszke et al., 2017) to implement our models with a Titan Xp GPU.
E XPERIMENTS
We apply our proposed RNAN to three classical image restoration tasks: image denoising, demosaicing, and compression artifacts reduction.
For image denoising and demosaicing, we follow the same setting as IRCNN (Zhang et al., 2017b).
For image compression artifacts reduction, we follow the same setting as ARCNN (Dong et al., 2015).
We use 800 training images in DIV2K (Timofte et al., 2017; Agustsson & Timofte, 2017) to train all of our models.
For each task, we use commonly used dataset for testing and report PSNR and/or SSIM (Wang et al., 2004) to evaluate the results of each method.
More results are shown in Appendix A.
[Table]
[Table]
4. A BLATION S TUDY
We show ablation study in Table 1 to investigate the effects of different components in RNAN.
Non-local Mixed Attention .
In cases 1, all the mask branches and non-local blocks are removed.
In case 4, we enable non-local mixed attention with same block number as in case 1.
The positive effect of non-local mixed attention is demonstrated by its obvious performance improvement.
Mask branch .
We also learn that mask branch contributes to performance improvement, no matter non-local blocks are used (cases 3 and 4) or not (cases 1 and 2).
This’s mainly because mask branch provides informative attention to the network, gaining better representational ability.
Non-local block .
Non-local block also contributes to the network ability obviously, no matter mask branches are used (cases 2 and 4) or not (cases 1 and 3).
With non-local information from low-level and high-level features, RNAN performs better image restoration.
Block Number .
When comparing cases 2, 4, and 7, we learn that more non-local blocks achieve better results.
However, the introduction of non-local block consumes much time.
So we usenon-local blocks by considering low- and high-level features.
When RNAB number is ﬁxed in cases 5 and 7 or cases 6 and 8, performance also beneﬁts from more RABs.
[Table] [Table]
4. C OLOR AND G RAY I MAGE D ENOISING
We compare our RNAN with state-of-the-art denoising methods: BM3D (Dabov et al., 2007b),
CBM3D (Dabov et al., 2007a), TNRD (Chen & Pock, 2017), RED (Mao et al., 2016),
DnCNN (Zhang et al., 2017a), MemNet (Tai et al., 2017), IRCNN (Zhang et al., 2017b), and FFDNet (Zhang et al., 2017c).
Kodak24 (http://r0k.us/graphics/kodak/), BSD68 (Martin et al., 2001), and Urban100 (Huang et al., 2015) are used for color and gray image denoising.
AWGN noises of different levels (e.g., 10, 30, 50, and 70) are added to clean images.
Quantitative results are shown in Tables 2 and 3.
As we can see that our proposed RNAN achieves the best results on all the datasets with all noise levels.
Our proposed non-local attention covers the information from the whole image, which should be effective for heavy image denoising.
To demonstrate this analysis, we take noise level σ = 70 as an example.
We can see that our proposed
RNAN achieves 0.48, 0.30, and 1.06 dB PSNR gains over the second best method FFDNet.
This comparison strongly shows the effectiveness of our proposed non-local mixed attention.
We also show visual results in Figures 4 and 5.
With the learned non-local mixed attention, RNAN treats different image parts distinctively, alleviating over-smoothing artifacts obviously.
[Table]
4. I MAGE D EMOSAICING
Following the same setting in IRCNN (Zhang et al., 2017b), we compare image demosaicing results with those of IRCNN on McMaster (Zhang et al., 2017b), Kodak24, BSD68, and Urban100.
Since
IRCNN has been one of the best methods for image demosaicing and limited space, we only compare with IRCNN in Table 4.
As we can see, mosaiced images have very poor quality, resulting in very low PSNR and SSIM values.
IRCNN can enhance the low-quality images and achieve relatively high values of PSNR and SSIM.
Our RNAN can still make signiﬁcant improvements over IRCNN.
Using local and non-local attention, our RNAN can better handle the degradation situation.
Visual results are shown in Figure 6.
Although IRCNN can remove mosaicing effect greatly, there’re still some artifacts in its results (e.g., blocking artifacts in ‘img 026’).
However, RNAN recovers more faithful color and alliciates blocking artifacts.
4. I MAGE C OMPRESSION A RTIFACTS R EDUCTION
We further apply our RNAN to reduce image compression artifacts.
We compare our RNAN with
SA-DCT (Foi et al., 2007), ARCNN (Dong et al., 2015), and DnCNN (Zhang et al., 2017a).
We apply the standard JPEG compression scheme to obtain the compressed images by following (Dong et al., 2015).
Four JPEG quality settings q = 10, 20, 30, 40 are used in Matlab JPEG encoder.
Here, we only focus on the restoration of Y channel (in YCbCr space) to keep fair comparison with other methods.
We use the same datasets LIVE1 (Sheikh et al., 2005) and Classic5 (Foi et al., 2007) in
ARCNN and report PSNR/SSIM values in Table 5.
As we can see, our RNAN achieves the best
PSNR and SSIM values on LIVE1 and Classic5 with all JPEG qualities.
We further shown visual comparisons in Figure 7.
We provide comparisons under very low image quality ( q =10).
The blocking artifacts can be removed to some degree, but ARCNN, TNRD, and DnCNN would also over-smooth some structures.
RNAN obtains more details with consistent structures by considering non-local mixed attention.
[Table]
4. I MAGE S UPER -R ESOLUTION
We further compare our RNAN with state-of-the-art SR methods: EDSR (Lim et al., 2017), SRMDNF (Zhang et al., 2018a), D-DBPN (Haris et al., 2018)
[Table]
and RCAN Zhang et al.
(2018b).
Similar to (Lim et al., 2017; Zhang et al., 2018c), we also introduce self-ensemble strategy to further improve our RNAN and denote the self-ensembled one as RNAN+.
As shown in Table
[Table]
our RNAN+ achieves the second best performance among benchmark datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001),
Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017).
Even without self-ensemble, our RNAN achieves third best results in most cases.
Such improvements are notable, because the parameter number of RNAN is 7.5 M, far smaller than 43 M in EDSR and 16 M in RCAN.
The network depth of our RNAN (about 120 convolutional layers) is also far shallower than that of RCAN, which has about 400 convolutional layers.
It indicates that non-local attention make better use of main network, saving much network parameter.
In Figure 8, we conduct image SR ( × 4) with several state-of-the-art methods.
We can see that our RNAN obtains better visually pleasing results with ﬁner structures.
These comparisons further demonstrate the effectiveness of our proposed RNAN with the usage of non-local mixed attention.
[Table]
4. P ARAMETERS AND R UNNING T IME A NALYSES
We also compare parameters, running time, and performance based on color image denoising in
Table 7.
PSNR values are tested on Urban100 ( σ =50).
RNAN with 10 blocks achieves the best performance with the highest parameter number, which can be reduced to only 2 blocks and obtains second best performance.
Here, we report running time for reference, because the time is related to implementation platform and code.
[Table]
C ONCLUSIONS
In this paper, we propose residual non-local attention networks for high-quality image restoration.
The networks are built by stacking local and non-local attention blocks, which extract local and non-local attention-aware features.
Speciﬁcally, there are trunk branch and (non-)local mask branch in each (non-)local attention block.
The trunk branches are used to extract hierarchical features.
The mask branches to adaptively rescale hierarchical features with soft weights.
We further generate nonlocal attention by considering the whole feature map, where the response at a position is obtained as a weighted sum of feature response at all positions.
Furthermore, we propose residual local and nonlocal attention learning to train very deep networks.
We introduce the input feature into attention computation, being more suitable for image restoration.
Our proposed RNAN achieves state-of-theart results in several image restoration tasks with moderate model size and running time.
[Table]
σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours) BSD68: HQ Noisy ( σ =50)
CBM3D TNRD
RED DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
BSD68:
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
[Figure] [Figure]
[Figure]
A A PPENDIX A. T RAIN RNAN WITH S MALL T RAINING D ATA
All the results shown in the main paper are based on DIV2K training data.
Here, we retrain our
RNAN on small training sets for 5 tasks.
In Table 8, for each task, we only refer the second best method from our main paper to compare.
For training data, we use BSD400 (Martin et al., 2001) for color/gray image denoising and demosaicing.
We use 91 images in Yang et al.
(2008) and 200 images in Martin et al.
(2001) (denoted as SR291) for image compression artifacts reduction and super-resolution.
FFDNet used BSD400 (Martin et al., 2001), 400 images from ImageNet Deng et al.
(2009), and 4,744 images in Waterloo Exploration Database Ma et al.
Here, ‘BSD400+’ is used to denote ‘BSD400+ImageNet400+WED4744’.
According to Table 8, we use the same or even smaller training set for our RNAN and obtain better results for 5 tasks.
These experiments demonstrate the effectiveness of our RNAN for general image restoration tasks.
[Table]
A. V ISUAL R ESULTS Color and Gray Image Denoising
We show color and gray image denoising comparisons in
Figures 9 and 10 respectively.
We can see that our RNAN recovers shaper edges.
Unlike most of other methods, which over-smooth some details (e.g., tiny lines), RNAN can reduce noise and maintain more details.
With the learned non-local mixed attention, RNAN treats different image parts distinctively, alleviating over-smoothing artifacts obviously.
Image Compression Artifacts Reduction (CAR) .
In Figure 11, we provide comparisons under very low image quality ( q =10).
The blocking artifacts can be removed to some degree, but ARCNN,
TNRD, and DnCNN would also over-smooth some structures.
In contrast, RNAN obatins more details with consistent structures by considering non-local mixed attention.
Image Super-Resolution (SR) .
In Figure 12, we conduct image SR ( × 4) with several state-of-theart methods, such as SRCNN (Dong et al., 2016), VDSR (Kim et al., 2016), LapSRN (Lai et al., 2017), MemNet (Tai et al., 2017), EDSR (Lim et al., 2017), SRMDNF (Zhang et al., 2018a), and DDBPN (Haris et al., 2018).
We can see that most of compared methods would suffer from distortion or output totally wrong structures.
For tiny line, edge structures, or some textures, they fail to recover and introduce blurring artifacts.
Instead, our RNAN obtains better visually pleasing results with ﬁner structures.
These comparisons further demonstrate the effectiveness of our proposed RNAN with the usage of non-local mixed attention.
Kodak24: kodim
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Kodak24: kodim
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN MemNet IRCNN FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
CBM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Kodak24: kodim
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Kodak24: kodim
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
BSD68:
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
BSD68:
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Urban100: img
HQ
Noisy ( σ =50)
BM3D
TNRD
RED
DnCNN
MemNet
IRCNN
FFDNet
RNAN (ours)
Classic5: boats
HQ
JPEG ( q =10)
ARCNN
TNRD
DnCNN
RNAN
LIVE1: buildings
HQ
JPEG ( q =10)
ARCNN
TNRD
DnCNN
RNAN
LIVE1: womanhat
HQ
JPEG ( q =10)
ARCNN
TNRD
DnCNN
RNAN
LIVE1: sailing
HQ
JPEG ( q =10)
ARCNN
TNRD
DnCNN
RNAN
LIVE1: paintedhouse
HQ
JPEG ( q =10)
ARCNN
TNRD
DnCNN
RNAN
LIVE1: sailing
HQ
JPEG ( q =10)
ARCNN
TNRD
DnCNN
RNAN
LIVE1: sailing
HQ
JPEG ( q =10)
ARCNN
TNRD
DnCNN
RNAN
[Figure]
SRCNN
VDSR
LapSRN
MemNet
EDSR
SRMDNF
D-DBPN
RNAN (
[Figure] [Figure]
Bicubic
SRCNN
VDSR
LapSRN
MemNet
EDSR
SRMDNF
D-DBPN
RNAN (ours)
[Figure]
Bicubic
SRCNN
VDSR
LapSRN
MemNet
EDSR
SRMDNF
D-DBPN
RNAN (ours)
[Figure]
Bicubic
SRCNN
VDSR
LapSRN
MemNet
EDSR
SRMDNF
D-DBPN
RNAN (ours)
[Figure]
Bicubic
SRCNN
VDSR
LapSRN
MemNet
EDSR
SRMDNF
D-DBPN
RNAN (ours)
[Figure]
VDSR
LapSRN
MemNet
EDSR
SRMDNF
D
[Figure]
DBPN
RNAN (
[Figure] [Figure]
HR
[Figure]
VDSR
LapSRN
MemNet
EDSR
SRMDNF
[Figure]
RNAN (
[Figure] [Figure]
LapSRN
[Figure]