Cross Aggregation Transformer for Image Restoration
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recently, Transformer architecture has been introduced into image restoration toreplace convolution neural network (CNN) with surprising results.
Consideringthe high computational complexity of Transformer with global attention, somemethods use the local square window to limit the scope of self-attention.
However,these methods lack direct interaction among different windows, which limits theestablishment of long-range dependencies.
To address the above issue, we proposea new image restoration model, Cross Aggregation Transformer (CAT).
The coreof our CAT is the Rectangle-Window Self-Attention (Rwin-SA), which utilizeshorizontal and vertical rectangle window attention in different heads parallellyto expand the attention area and aggregate the features cross different windows.
We also introduce the Axial-Shift operation for different window interactions.
Furthermore, we propose the Locality Complementary Module to complement the self-attention mechanism, which incorporates the inductive bias of CNN ( e.g. ,translation invariance and locality) into Transformer, enabling global-local coupling.
Extensive experiments demonstrate that our CAT outperforms recent state-of-theart methods on several image restoration applications.
Introduction
Image restoration is a long-term low-level vision problem, aimed at recovering high-quality (HQ)images from low-quality (LQ) counterparts.
Depending on the type of degradation, there are manysub-problems, such as super-resolution (SR), image denoising and JPEG compression artifactreduction.
In recent decades, many researchers have proposed various theories and methods to solvethis problem, e g. , inverse ﬁltering and wiener ﬁltering.
Meanwhile, with the increase of computingresources, neural networks have become the mainstream of image restoration, especially convolutionalneural network (CNN) [9, 10, 43], which achieve impressive performance over traditional methods.
Recently, some researchers have been using Transformer proposed in natural language processing(NLP) to replace CNN and achieving excellent performance in multiple vision tasks, such as imageclassiﬁcation [12, 34, 11], semantic segmentation [37], and object detection [38].
The core component of Transformer is the multi-head self-attention (MHSA), which can effectively model long-range dependencies, while CNN is more inclined to extract local features.
Although Transformer caneffectively handle dependencies between tokens, its complexity is quadratic with the image resolution
( H × W ), i.e. , O (( HW ) 2 ) , which limits its application to high-resolution image tasks (includingimage restoration).
In high-level vision, hierarchical architecture [34, 22, 6] is applied to decreasecomputational complexity by down-sampling the feature map at each stage.
However, in imagerestoration, downsampling will cause the image to lose much information, hindering high-quality image reconstruction.
To address this problem, some methods [4, 20, 40] propose to divide the image into smaller independent patches ( e.g. , 8 × 8) and perform attention separately or apply selfattention (SA) in the feature dimension instead of the spatial dimension, thus achieving linear complexity.
However, the square window lacks inter-window interaction, leading to the slow increaseof the receptive ﬁeld.
Also, the channel-wise attention mechanism loses a large amount of spatialinformation.
All these problems restrict the performance of Transformer in image restoration.
To solve the above problems of Transformer application in image restoration tasks, we proposea new image restoration Transformer model named Cross Aggregation Transformer (CAT).
Our
CAT utilizes local window self-attention to restrict the computational complexity and aggregatesfeatures cross different windows to expand the receptive ﬁeld.
The key part of our CAT is a newself-attention mechanism, named Rectangle-Window Self-Attention (Rwin-SA).
Speciﬁcally, the
Rwin-SA performs the attention operation in a separate rectangle window ( sh (cid:54) =
sw ) rather than thesquare ( sh = sw ) one, where sh and sw are window height and width.
And we divide multiple headsinto two groups and perform horizontal ( sh < sw ) and vertical ( sh > sw ) rectangle window self-attentionin parallel.
This method effectively expands the attention area and aggregates different features inhorizontal and vertical directions while keeping the computational complexity unchanged.
Moreover,inspired by CSwin [11], which performs cross-shaped window self-attention, we ﬁx the length of oneside of the rectangle window to be H or W (image resolution). Then the window becomes a stripealong the axis, called axial rectangle window.
It can generate more window interactions and obtain alarger attention area than the regular rectangle window.
We also propose a new shift operation named Axial-Shift Operation , which is applied betweenconsecutive Rwin-SA, to further increase the interaction of the different windows.
Like our Rwin-SA,axial-shift is also divided into horizontal and vertical directions, which acts on the correspondingwindow respectively.
Compared with the shift operation in Swin Transformer [22], our axial-shiftexplicitly realizes the interaction between horizontal-horizontal or vertical-vertical windows, andimplicitly enables the interaction between horizontal and vertical windows.
Furthermore, we propose Locality Complementary Module (LCM), a convolution operation thatit performed on the value ( V ) in the self-attention mechanism in parallel with the attention part.
Compared with Transformer that mainly builds global long-range dependencies, CNN has thecharacteristics of translation invariance and locality, and can effectively capture 2D local structures ofthe image ( e.g. , corners and edges).
The LCM can complements Rwin-SA with locality information,enabling the coupling of global (self-attention) and local (convolution) information.
Based on the above three novel designs, our CAT can realize the feature aggregation cross differentwindows, thus possessing the long-range dependencies modeling capability.
We apply our CAT totwo classic image restoration tasks: image SR and JPEG compression artifacts reduction.
Extensiveexperiments demonstrate the superiority of our method.
Our main contributions are as follows
• We propose a new image restoration Transformer model named Cross Aggregation Trans- former (CAT).
The CAT utilizes the window self-attention and aggregates the features cross different windows, thus achieving a large receptive ﬁeld with linear complexity.
• We propose a novel self-attention mechanism, named Rwin-SA, using rectangle windowself-attention with axial-shift operation.
Moreover, we propose the locality complementarymodule to realize the coupling of global and local information.
• We apply our CAT to classic image restoration tasks: image SR and JPEG compression artifact reduction.
Extensive experiments show that our proposed model achieves state-of- the-art performance both quantitatively and visually.
Related Work
Image Restoration.
CNNs have been achieving impressive performance over the traditional approaches and becoming the mainstream method in the ﬁeld of image restoration since Dong etal.
proposed SRCNN [10] and ARCNN [9] for image SR and JPEG compression artifact reduction, respectively.
Currently, many efforts have been devoted to building very deep CNNs toimprove performance through multiple methods, e.g. , residual learning strategies [14], dense connection [15], dropout [18], and UNet architecture [30].
Meanwhile, spatial and channel attentionmechanisms [44, 28, 45, 19, 5] are utilized to make the network more focused on speciﬁc informationin the feature space.
However, most CNN-based methods are still hard to capture global dependencies.
Vision Transformer.
Vision Transformer (ViT) achieves surprising results in high-level visiontasks [12, 34, 37, 38]. Meanwhile, many efﬁcient attention mechanisms have been proposed toimprove the performance of ViT. Swin Transformer [22] uses local window attention and realizes the
[Figure]
interaction between windows through shift operations.
CSwin [11] proposes cross-shaped windowattention, and Twins [6] adopt a combination of global and local attention.
Inspired by the successof Transformer in high-level tasks, some works have been trying to apply Transformer to low-levelvision tasks, such as IPT [4].
However, employing original ViT [12] for restoration tasks is hard sincethe complexity of full self-attention is quadratic to image size.
Some methods have been proposedto solve this problem.
SwinIR [20], based on Swin Transformer, utilizes an 8 × 8 window to dividethe image and executes self-attention in each window separately.
It also uses the shift operation. Uformer [36] also applies 8 × 8 local window and introduces UNet architecture to capture much moreglobal dependencies.
Restormer [40] calculates cross-covariance across channel dimensions ratherthan spatial ones, making the complexity of SA linear with image resolution.
However, the 8 × 8square window limits the receptive ﬁeld of the Transformer even with the shift operation.
And thechannel-wise attention mechanisms may cause spatial information loss.
Furthermore, above methodsdo not effectively combine convolution with attention mechanisms, thus limiting model capabilities.
Method
[Figure]
To build an efﬁcient Transformer model suitable for image restoration, we propose a rectanglewindow self-attention (Rwin-SA) to replace vanilla multi-head self-attention mechanism in ViT [12]and form the cross aggregation Transformer block (CATB).
Then we use RCAN [44] as the backboneand replace the residual channel attention block (RCAB) proposed in RCAN with our CATB to buildthe model cross aggregation Transformer (CAT), which is illustrated in Fig.
We ﬁrst describethe overall structure of CAT and then introduce the three core designs of the CATB, includingrectangle-window self-attention, axial-shift operation, and locality complementary module.
3.1 CAT Architecture
As shown in Fig.
1, just like RCAN, our CAT consists of three modules: shallow feature extraction,deep feature extraction, and reconstruction part.
For a low-quality input image I LQ ∈
R H × W × C in ,we leverage one convolution layer as shallow feature extraction to get the low-level feature
F 0 ∈ R H × W × C , where H, W, C in , C are the image height, width, input channel, and the featurenumber, respectively.
Then, the shallow feature is processed by the deep feature extraction module,which is composed of N 1 residual groups (RG) and one convolution layer, and extracted the deep feature (denoted as F DF ∈
R H × W × C ).
The convolution layer here is employed to aggregate previouslyextracted features from RG.
Moreover, a residual connection is applied in the deep feature extractionpart to make the training stable.
We use CATB to replace RCAB in RG, as we mentioned above.
So,
RG consists of N 2 cross aggregation Transformer blocks and a convolution layer that can introducethe unique properties (translation invariance and locality) of CNN into the output of Transformerblocks.
In addition, a residual strategy is also applied in each RG.
Finally, we can get the high-quality output image I HQ ∈
R H
× W × C out through the reconstruction part,where C out is the output dimension.
And following SwinIR [20], the composition of modules isdifferent for different image restoration tasks.
More speciﬁcally, for image SR, a sub-pixel convolutionlayer [32] is used to upsample the deep feature F DF to the same size of the high-resolution outputand there is a convolution layer before and after the upsampling module to aggregate the features.
For JPEG compression artifact reduction, the reconstruction is just a convolution layer to adjust thechannel dimension of the F DF from C to C out .
The low-quality input I LQ is then added to theconvolution output to produce a high-quality output I HQ .
This residual learning between low-qualityimages and high-quality images accelerates network model convergence.
[Figure]
𝑠ℎ 𝑠𝑤 𝑠𝑤 𝑠ℎ 𝑠𝑤/ 𝑠ℎ/ 𝑠ℎ/2 𝑠𝑤/
[Figure]
3.2 Cross Aggregation Transformer Block
The cross aggregation Transformer block (CATB) is the key part of our CAT, shown in Fig.
1, which ismore suitable for image restoration tasks than the previous Transformer structures [20, 40].
It utilizesa new attention mechanism with three novel designs: rectangle-window self-attention, axial-shiftoperation, and locality complementary module.
We will describe these parts in detail below.
Rectangle-Window Self-Attention .
To alleviate the drawbacks of previous methods [4, 20, 40], wepropose a new window attention mechanism as shown in Fig. 2(a), called rectangle-window selfattention (Rwin-SA).
Rwin-SA uses the rectangle window ( sh (cid:54) =
sw ) rather than the square ( sh = sw )one, where sh and sw represent the height and width of the rectangle.
Moreover, we divide therectangle window into the horizontal one ( sh < sw , denoted as H-Rwin) and the vertical one ( sh > sw ,denoted as V-Rwin), and employ them to different attention heads in parallel.
Speciﬁcally, given the input X ∈ R H
× W × C , we execute the attention operation M (the heads number)times in parallel.
For each attention head, we split X into non-overlapping sh × sw rectangle windowsand denote the i -th rectangle window feature as X i ∈ R
( sh × sw ) × C , where i = 1 , . . .
, H × W sh × sw .
Then theself-attention of X i for the m -th head can be computed as
[Equation]
where Y mi ∈ R ( sh × sw ) × D is the attention feature of X i in the m -th head, and Q im , K
im ,
V im ∈ R C × drepresent the projection matrices of query, key, and value for m -th head.
D =
d = CM is the channeldimension in each head and B is the dynamic relative position encoding [35].
Performing the attentionoperation on all X i ( i =1 , . . .
, H × W sh × sw ) and reshaping and merging them in the order of division, wecan get the attention feature Y m ∈
R H
× W × D of X .
In general, the above attention operation is thesame for H-Rwin and V-Rwin, where the sh and sw take different values.
Assuming that the number of attention heads M is even, we divide the heads equally into two partsand execute H-Rwin for the ﬁrst part and V-Rwin for the second part in parallel.
Finally, the outputsof two parts are concatenated along the channel dimension.
The process is formulated as
[Equation]
where Y 1 , . .
, Y M 2 are the output of the head using H-Rwin, and Y M 2 +1 , . . .
, Y M are from the
V-Rwin.
The W p ∈ R C × C represents the projection matrix for feature fusion.
As shown in Fig.
3,through H-Rwin and V-Rwin, we can aggregate features cross different windows and expand theattention area without increasing computational complexity.
Moreover, the rectangle window cancapture different features in horizontal and vertical directions for each pixel, which is hard for squarewindow to handle with comparable computation resource.
This property is crucial for some datasets( e.g. , Urban
[Figure]
W
[Figure]
Furthermore, as shown in Fig.
3, inspired by CSwin [11], we ﬁx thelength of one side of the rectangle tobe the image resolution H or W (theother side length denoted as sl ). Thenthe window becomes a stripe alongthe axis, called axial rectangle window (denoted as axial-Rwin).
Sincethe axial window changes as the image changes, the axial-Rwin is moreﬂexible than the regular rectangle window, which side length is [ sh, sw ] or [ sw, sh ] (denoted asregular-Rwin).
The computational complexity of regular-Rwin and axial-Rwin are
[Equation]
When H = W (square input), the complexity of regular-Rwin is O
( H 2 ) and axial-Rwin is O
( H 3 ) .
Although axial-Rwin is more complex than regular-Rwin, it is still smaller than full-attention ( O ( H 4 ) ),and applicable for high-resolution images.
And axial-Rwin has a larger attention area than that inregular-Rwin, which can capture more information, especially on the axial direction.
Axial-Shift Operation .
To further expand the receptive ﬁeld of Rwin-SA so that each pixel canaggregate more information, we propose a new shift operation called axial-shift, for Rwin-SA.
Asshown in Fig.
2(b), axial-shift includes two shift operations, the horizontal one is called H-Shift andthe vertical one is called V-Shift, corresponding to H-Rwin and V-Rwin, respectively.
The axial-shiftoperation moves the window partition down and left by sh 2 and sw 2 pixels, where sh and sw are thewindow height and width of H-Rwin and V-Rwin.
In the implementation, we cyclically shift thefeature map to down-left direction.
Then we execute H-Rwin and V-Rwin on corresponding shiftedfeature maps and use a masking mechanism to avoid false interactions between non-adjacent pixels.
The axial-shift operation can be regarded as a more general shifted window operation in Swin
Transformer [22].
When sh and sw in H-Rwin and V-Rwin are all the same, it degenerates intothe swin shift operation.
Compared with the swin shift operation, our axial-shift enables a broaderrange of window interaction: (1) H(V)-Rwin with H(V)-Rwin .
Intuitively, the new window partitionenables the interactive fusion of different rectangle window of the same type (horizontal or verticalone), just like the swin shift operation.
(2) H-Rwin with V-Rwin .
The Rwin-SA performs H-Rwinand V-Rwin simultaneously and fuses them through the projection in Eq.
For this reason,axial-shift can implicitly implement interaction between horizontal and vertical rectangle windows.
MLP
LayerNorm
Rwin - SA
LayerNorm
Rwin - SA
LEB
Projection
Q
K V Rwin-SA LEB
Projection
Q K
V
Rwin-SA
LCM
Locality Complementary Module .
Transformer can effectively captureglobal information and model long-range dependencies between pixels.
However, the inductive bias (translation invariance and locality) of CNN are stillindispensable in image restoration tasks.
It can aggregate local features andextract the basic structures of an image ( e.g. , corners and edges).
In order tocomplement the Transformer with the locality and achieve global and localcoupling, we propose to add an independent convolution operation when calculating self-attention, called locality complementary module.
As shown in
Fig.
4, different from the previous practice in Transformer [39, 7], we directlyoperate convolution on value V .
The process is formulated as
[Equation]
where Y 1 , . .
, Y M , and W P are the same as Eq.
The V ∈ R H × W × C isthe value projected directly from X without window partition ( V im is the partition of V in m -thattention head deﬁned in Eq.
(2)), and Conv( · ) represents the 3 × 3 depth-wise convolution.
This operation has two advantages over executing convolution sequentially or using convolution on
X : (1) Using convolution as a parallel module allows Transformer block to adaptively choose toadopt attention or convolution operations, which is more ﬂexible than sequential execution; (2) From
Eq.
(5), we can ﬁnd that self-attention can be regarded as a content-dependent and dynamic weightacting on V .
The convolution operation is equivalent to a learnable and static weight.
Therefore, theconvolution operation on V is conducted in the same feature domain as the attention operation.
[Table]
Cross Aggregation Transformer Block .
CATB is composed of Rwin-SA and MLP, which has twolinear projection layers with a GELU non-linearity between them.
We formulate the block as
[Equation]
where X in and X out are the input and output feature maps of CATB; LN( · ) is the LayerNormoperation.
And axial-shift operation is used on the interval between two consecutive cross aggregation
Transformer blocks to enhance the interaction among different windows.
Experiments
4.1 Experimental Settings
Data and Evaluation.
For image SR, we choose DIV2K [33] and Flickr2K [21] as the trainingdata.
And benchmark datasets for evaluation include Set5 [2], Set14 [41], B100 [24], Urban100 [16],and Manga109 [25] with three upscaling factors: × 2, × 3, and × 4.
The low-resolution images aregenerated by Bicubic downsampling.
For JPEG compression artifact reduction, training set consistsof DIV2K [33], Flickr2K [21], BSD500 [1], and WED [23].
And we have two testing datasets:
Classic5 [13] and LIVE1 [31], with JPEG compression qualities of 10, 20, 30, and 40.
We use themetrics PSNR and SSIM [23] on the Y channel of the YCbCr space to evaluate the results.
Implementation Details.
We set the residual group (RG) number as N 1 =6 and the cross aggregation
Transformer block (CATB) number as N 2 =6 for each RG.
The channel dimension, attention headnumber, and MLP expansion ratio for each CATB are set as 180, 6, and 4, respectively.
For image SR,we build two versions of CAT, called CAT-R and CAT-A. CAT-R uses the regular rectangle window(regular-Rwin), and CAT-A uses the axial rectangle window (axial-Rwin).
The regular-Rwin and theaxial-Rwin are deﬁned in Sec. 3.2.
For CAT-R, we set [ sh, sw ] as [4 , 16] (or [16 , 4] ) for each CATBin each RG, and for CAT-A, we set sl as [2 , 2 , 2 , 4 , 4 , 4] for different residual groups.
For JPEGcompression artifact reduction, we also employ CAT-A with the same sl setting as in image SR.
Training Settings.
For image SR, we train the model with batch size 32, where each input image is randomly cropped to 64 × 64 size, and the total training iterations are 500K.
We adopt
Adam optimizer [17] with β 1 = 0 .
9 and β 2 = 0 .
99 to minimize the L 1 pixel loss following thework [44, 45, 20].
The initial learning rate is set as 2 × 10 − 4 and reduced by half at the milestone [250K,400K,450K,475K].
For JPEG compression artifact reduction, the batch size and totaliterations are 8 and 1600K, where the input image size is 128 × 128.
Same as image SR, we adopt
Adam optimizer to minimize Charbonnier Loss [3] with (cid:15) = 10 − 3 , and the learning rate is initialized to2 × 10 − 4 , which is halved at [800K,1200K,1400K,1500K].
Furthermore, we randomly utilize rotationand ﬂipping to augment the training data for image SR and JPEG compression artifact reduction.
Weuse PyTorch [29] to implement our models with 4 Tesla V100 GPUs.
4.2 Ablation Study
For the ablation study, we use the dataset DIV2K [33] and Flickr2K [21] to train CAT on the image
SR ( × 2) task, where the input image patch size is 64 × 64 and the iterations are 150K.
The testingdataset is Urban100 [16].
FLOPs are calculated on input image size 3 × 128 × 128.
Rectangle-Window Self-Attention .
Table 1a demonstrates that rectangle-window self-attention ismore effective than the square window attention ( e.g. , SwinIR [20]).
With no increase in computational complexity ( e.g. , FLOPs), the rectangle window can improve the performance from 32.
[Table]
dB to 32.66 dB compared to the square one.
Moreover, when we adopt the axial-shift operation in
Rwin-SA, PSNR can reach 32.91 dB. And the square window attention of the swin shift operationonly obtains 32.75 dB. It can be found that our Rwin-SA can aggregate more information and is moresuitable for image restoration than attention mechanisms in Swin Transformer [22].
Locality Complementary Module .
Table 1b shows the impact of locality complementary module on
CAT-R (regular-Rwin) and CAT-A (axial-Rwin).
With the LCM, CAT-R and CAT-A can achieve 0.07dB and 0.10 dB gains over the cases without LCM, respectively.
These results show that the inductivebias of CNN can effectively improve the performance of Transformer in image restoration.
Andcomparing the complexity, it can be found that LCM only increases FLOPs by 0 .
26% ~ 0 .
32% , whichwill not affect the model efﬁciency too much.
Furthermore, LCM is embedded into self-attention as anew module without changing the implementation of the attention part.
Window Size Impact .
From above analyses, we know that our Rwin-SA is more suitable for imagerestoration tasks than the swin attention mechanism.
We further investigate the impact of differentwindow sizes.
The results are provided in Table 1c.
Comparing regular-Rwin (CAT-R) and axial-Rwin(CAT-A), we can ﬁnd that the performance of axial-Rwin is worse than regular-Rwin (32.97 dB vs.32.98 dB), when the side of axial-Rwin is small ( sl = 2 ), despite its higher complexity (323.5G vs.282.7G).
It may be because the side of axial-Rwin ( sl ) is too narrow to capture enough informationand may even introduce noise.
Therefore, the result of axial-Rwin rises rapidly as sl increase.
Theseresults show that both side length and window size of Rwin are crucial for performance.
4.3 Image Super-Resolution
We compare our two models CAT-R and CAT-A with state-of-the-art methods: EDSR [21],
RCAN [44], SAN [8], IGNN [47], HAN [28], CSNLN [27], NLSA [26], IPT [4], and SwinIR [20].
Following the previous work [20, 44], we use self-ensemble strategy in testing and mark the modelwith a symbol “+”.
We use CAT-A for visual comparisons, abbreviated as CAT.
Urban100: img_024 (
[Figure]
HQ
Bicubic
EDSR [21]
RCAN [44]
SAN [8]
IGNN [47]
HAN [28]
CSNLN [27]
SwinIR [20]
CAT (ours)
Urban100: img_074 (
[Figure]
HQ
Bicubic
EDSR [21]
RCAN [44]
SAN [8]
IGNN [47]
HAN [28]
CSNLN [27]
SwinIR [20]
CAT (ours)
Urban100: img_092 (
[Figure]
HQ
Bicubic EDSR [21]
[Figure] [Figure]
IGNN [47]
HAN [28]
CSNLN [27]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
Quantitative Results.
Table 2 shows PSNR/SSIM comparisons for × 2, × 3, and × 4 image SR.
As we can see, our CAT-R (regular-Rwin) and CAT-A (axial-Rwin) signiﬁcantly outperform othermethods on all datasets with all scale factors.
Among these ﬁve datasets, CAT-R and CAT-A havethe most noticeable improvement in Urban100.
Compared with the recent Transformer model ( e.g. ,
SwinIR), for CAT-R, the PSNR gain reaches 0.27 dB for scale factor 2, and for CAT-A, the PSNRincreases by 0.45 dB. It shows that our CAT can capture more global information than previous
CNN-based and Transformer-based models, which is very effective for images in Urban100 witha large number of repetitive texture structures.
Moreover, compared with CAT-R, CAT-A yields0 . 18 ~ 0 .
27 dB improvements on Urban100.
All these results indicate the effectiveness of our method.
Visual Results.
We show visual comparisons ( × 4) in Fig.
We can observe that most comparedmethods can hardly recover accurate textures and suffer from blurring artifacts in some challengingcases.
In contrast, our CAT can reconstruct more high-frequency structural details and alleviate theblurring artifacts.
For example, in img_092, our CAT can recover correct and sharp textures, whilemost compared methods fail to recover image details.
These results demonstrate that our CAT hasmore robust representational ability to recover structural contents and textural details.
It also showsthat, compared with other models, our CAT is more effective in modeling long-range dependencies,which are crucial for reconstructing high-quality images.
4.4 JPEG Compression Artifact Reduction
We compare our CAT with state-of-the-art image compression artifact removal methods: RNAN [45],
RDN [46], DRUNet [42], and SwinIR [20].
Here, we only focus on the restoration of Y channel(in YCbCr space).
We also use self-ensemble strategy and mark the model with a symbol “+”.
Quantitative and visual comparisons are shown in Table 3 and Fig. 6, respectively.
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Quantitative Results.
Table 2 shows quantitative comparisons with four JPEG quality settings: 10,20, 30, and 40.
Our CAT+ performs better than other compared methods on LIVE1 and Classic5with all JPEG qualities.
Even without self-ensemble, our RCAN also outperforms other comparedmethods, except for PSNR on Classic5 ( q =10).
Especially when q = 40 , CAT achieves 0.05 and 0.06dB improvements over the Transformer-based model SwinIR.
Even in a very low compression quality( e.g. , 10), CAT achieves 29.89 dB in LIVE1, higher than previous methods.
Visual Results.
We show visual comparisons at very low image quality ( q =10) in Fig.
For previousmethods, RNAN [45], RDN [46], and SwinIR [20], blocking artifacts are hardly to be totally removedand some structures are over-smoothed in some challenging cases.
In contrast, our CAT can recovermore details and remove blocking artifacts, resulting in sharper edges and more explicit textures.
These visual comparisons further demonstrate the effectiveness of our proposed CAT.
4.5 Model Size Analyses
Table 4 shows the comparison of performance, computational complexity ( e.g. , FLOPs), and parameter numbers on image SR.
FLOPs are measured when output size is set to 3 × 512 × 512 and PSNRvalues are tested on Urban100 ( × 4).
Although our CAT achieves excellent performance, it has lesscomputational complexity and parameters than EDSR [21].
And the computational complexity ofour CAT is much lower than CSNLN [27].
Compared to other CNN-based and Transformer-basedmethods, our CAT still has a comparable complexity and model size.
To demonstrate the effectivenessof our method, we provide another variant of CAT (CAT-R-2) with similar computational complexityand parameters to SwinIR. Our CAT-R-2 still outperforms SwinIR and other state-of-the-art methods. More details about CAT-R-2 are provided in the supplementary material.
Conclusion
In this paper, we propose a new Transformer model named cross aggregation Transformer (CAT)for image restoration with the rectangle-window self-attention (Rwin-SA) mechanism as the keypart.
Speciﬁcally, Rwin-SA performs H-Rwin and V-Rwin attention parallelly in different heads.
This mechanism can aggregate the features cross different windows and increase the receptive ﬁeldwithout additional computational cost.
Meanwhile, we propose an axial-shift operation accordingto the property of rectangle-window to further fuse different windows, which complements RwinSA.
Furthermore, the locality complementary module (LCM) introduces locality into self-attention,adding the inductive bias of CNN to Transformer.
Through the usage of LCM, Rwin-SA can realizethe coupling of global and local information, which is more conducive to image restoration.
Extensiveexperiments on image SR and JPEG compression artifact reduction demonstrate that our proposed
CAT outperforms current state-of-the-art methods.
In future work, we believe our method can be alsoapplicable to other
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See the supplementary(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See the supplementary(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec 3.2(b)
Did you include complete proofs of all theoretical results?
[Yes] See Sec 3.23.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] See the supplementary
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Implementation Details in Sec.
4.1 and Training Settings in Sec. 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Training Settings in Sec. 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[Yes] The used data is all publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
infor-
mation or offensive content?
[Yes] The used data doesn’t contain any personally identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]