Paper: Xformer: Hybrid X-Shaped Transformer for Image Denoising (ICLR-2024)

Reviewer comment (Experiments):
The authors claim that "we make the last encoder involving STBs of two branches share parameters for the purpose of computational efficiency." However, it is unclear how much the performance will be influenced by the parameter-sharing strategy. It is also not clear why it is critical to share parameters for this place in the network. Why not share parameters in other places?

Author response:
**(1) The performance is not influenced by the parameter-sharing strategy.** In our work, we use the parameter-sharing strategy in the last encoder of U-Net structure network to reduce the model complexity and improve the computing efficiency. During our study, we validate that this strategy has no negative influence on the model performance. We construct two types of neural networks. One is using the parameter-sharing strategy and the other is not. We conduct fair experimental comparison on the gaussian color image denoising task with noise level 15. We keep the same experimental settings. We show the experimental results here.
| PSNR results (dB)| Params (M) |FLOPs (G)| CBSD68 | Kodak24 | McMaster| Urban100 |
|-|-|-|-|-|-|-|
|w/o params-sharing|37.18|40.07|32.42|35.38|35.65|35.20|
|w/ params-sharing|25.07|39.11|34.42|35.37|35.65|35.22|

According to the table results above, **we demonstrate that the parameter-sharing strategy has little influence on the model performance**.

**(2) Explain the position of parameter-sharing strategy.** We choose to use the parameter-sharing strategy at the lowest layer of U-Net network. On the one hand, the module at the lowest layer processes the feature graph with the lowest resolution, so **the computational complexity is low when calculating self-attention interaction.** On the other hand, we use the STB to replace CTB by using parameter-sharing since the input feature of the lowest layer includes enough global information. Therefore, the STB can also capture the global feature by conducting self-attention interactions in the spatial dimension. **In short, the best position to using parameter-sharing strategy is just the last encoder of U-Net network.**

Reviewer comment (Experiments):
The authors claim that "In order to introduce contextualized information into self-attention computation, we choose to use 3×3 depth-wise convolution (Conv) following 1×1 Conv to generate query (Q), key (K), and value (V)." Why not directly use a vanilla 3×3 convolution?

Author response:
Thanks for your comments. We choose to use 3×3 depth-wise Conv and 1×1 Conv because we want to reduce the computational cost and model parameters. In practice, it is called the depthwise separable convolution and has been widely used in recent works. Therefore, we do not use a vanilla 3×3 convolution.

Reviewer comment (Experiments):
The authors claim that "Specifically, we use a 3×3 depth-wise convolution layer to refine the deep features from the spatial-wise branch for the purpose of saving computational consumption." Why not also using the 3×3 depth-wise convolution layer to refine the deep features from the channel-wise branch? Will it influence the performance compared to using the 3×3 vanilla convolution?

Author response:
Thanks for your questions. We give our replies as follows.

(1) Firstly, since the spatial information of features passed to STB is important, we use vanilla convolutions to process the deep features from the channel-wise branch. **It is because the vanilla convolution has more learnable parameters and thus extract more spatial information.** Therefore, we choose to use the 3×3 vanilla convolution.

(2)Secondly, we are going to provide specific experimental results to demonstrate the necessity of using 3×3 vanilla convolution. We train the new network named Xformer-change by replacing the 3×3 vanilla convolution in BCU with 3×3 depth-wise convolution on the gaussian color denoising task with noise level 15. We compare it to our current model Xformer. We keep the same experiment settings while training these two compared models. For ablation study, We train these models for 150k iterations with batch size 16. We report the compared results here.
| PSNR results (dB)| Params (M) |FLOPs (G)| CBSD68 | Kodak24 | McMaster| Urban100 |
|-|-|-|-|-|-|-|
|Xformer-change|24.71|40.90|34.27|35.15|35.25|34.76|
|Xformer|25.23|42.24|34.38|35.41|35.55|35.08|

As we can see, the model Xformer-change without using 3×3 vanilla convolution has limited performance. **Therefore, we demonstrate that using 3×3 vanilla convolution in BUC is necessary.**

Reviewer comment (Novelty):
The technical novelty is incremental. There are two core designs: the dual-branch architecture and the bilateral interactions between two branches, which are both typical designs and have been extensively explored in other work. Thus, the technical novelty is limited, especially compared to SwinIR and Restormer.

Author response:
Thanks for your comments. We consider your concern carefully and give our replies as follows.

**(1) The technical novelty is well.**  (a) At first, we should know that **the technical novelty should be consistent with the research motivation and background**. According to our research, creating a dual-branch network using different Transformer blocks is not common in the low-level vision field. (b) Besides, our motivation is not to validate the effectiveness of the dual-branch network. We want to explore stronger learning representation ability by utilizing different types of self-attention mechanisms in a unity neural network. **Although the dual-branch architecture and the bilateral interactions are not the first try, they have not been investigated to the exploit the advantages of both spatial-wise self attention and channel-wise self attention.** Therefore, our core designs are novel.

**(2) Compared to SwinIR and Restormer.** (a) We are willing to discuss the differences between our proposed method and these two works while considering the research motivation. In fact, it is important to further understand our motivation. In our work, the research subjects are the core self-attention mechanisms, but not the specific modules which have been proposed in SwinIR and Restormer. (b) **Our starting point is not to improve existing Transformer blocks for better performance.** In other words, we do not focus on using which kind of self-attention module proposed by previous works. **We consider more improvements from the level of feature capture and interaction.** Since existing Transformer-based denoising methods consider feature extraction from a single dimension, spatial or channel-wise, our method creatively uses a concurrent network to model global information from two dimensions. Besides, our proposed BCU is proved to effectively fuse two types of features. Extensive experimental results validate that our method has promising denoising performance. Here, we provide the compared results with SwinIR and Restormer on gaussian color image denoising.
| PSNR results (dB)| CBSD68 (15/25/50) | Kodak24 (15/25/50) | McMaster (15/25/50) | Urban100 (15/25/50) |
|-|-|-|-|-|
| SwinIR | 34.42/31.78/28.56| 35.34/32.89/29.79| 35.61/33.20/30.22| 35.13/32.90/29.82|
| Restormer| 34.40/31.79/28.60| 35.35/32.93/29.87| 35.61/33.34/30.30| 35.13/32.96/30.02|
| Xformer (ours)| **34.43/31.82/28.63** | **35.39/32.99/29.94** | **35.68/33.44/30.38** | **35.29/33.21/30.36** |

As we can see, our Xformer achieves the best performance on all benchmark datasets across three noise levels. It validates that the technical novelty brings Xformer the obvious performance improvement.

**(3)Conclusion.** In our work, **the core issues we want to solve are about the extraction and fusion of different types of features from spatial or channel dimensions.** Therefore, we design Xformer to improve the global information modeling from two dimensions. We conduct extensive experiments to demonstrate its promising denoising performance. **From this perspective, our work does not lack novelty.**

Reviewer comment (Evaluation):
Compared to Restormer, Xformer has limited performance improvement, especially on real image denoising scenarios which is more important for evaluation.

Author response:
Thanks for your comments. We propose Xformer and evaluate its performance on the synthetic and real-world image denoising tasks. Our replies are as follows.

**(1) About the performance on the synthetic datasets.** (a) Compared to Restormer, Xformer achieves obviously better performance on the synthetic image denoising tasks. For example, our Xformer obtains 0.34 dB higher PSNR score over Restormer on Urban100 for gaussian color image denoising with noise level 50. (b) Besides, we provide extensive visual results in the main paper and supplementary material to validate the superior of Xformer. **Therefore, Xformer has obvious performance improvement on the synthetic image denoising tasks.** 

**(2) About the performance on real image denoising.** (a) The commonly-used testing datasets on real image denoising task include SIDD and DND. We should attach importance to both of them. In fact, **the performance of Xformer on DND is much better than Restormer.** Our Xformer obtains 0.16dB higher PSNR score over Restormer on DND. (b) Furthermore, **the performance of Xformer on SIDD is influenced by the characteristics of the data itself.** As we know, the test process on SIDD is based on 1280 image patches of size 256×256 pixels from 40 high-resolution images. Since we carefully investigate these 1280 image patches, we find that they have very smooth texture features and most of them are from the background with unity color. **The evaluation on these images has limited ability to mine the strengths of our model.** It is because that our Xformer shows the strengths by combine the local fine-grained features and global features across channels. Therefore, the testing results on SIDD of Xformer is slightly below the results of Restormer since Restormer does not consider the feature extraction on spatial dimension.

**(3) Conclusion.** We explain the reasons of the limited performance of Xformer on SIDD dataset. According to the analyses above, **we confirm that our Xformer has obvious performance improvement over Restormer.**