Paper: Awakening Collective Wisdom: Elevating Super-Resolution Network Generalization through Cooperative Game Theory (ICLR-2024)

Reviewer comment (Experiments):
In Figure 1, the authors claim the proposed strategy alleviates the co-adaption phenomenon and achieve consistent channel response distribution. Can the dropout operation achieve the similar effect as the proposed strategy?

Author response:
**About dropout operation**

The conclusions draw from the previous work [1] indicates that employing dropout only at the last convolutional layer improves generalization performance, while even causing drop in performance at other layers. 
Therefore, we conducted a statistical analysis of channel responses in the $2^{nd}$ and last output layers of SRResNet, SRResNet with dropout, and SRResNet-EATS (all trained on the Five5K dataset) across 100 randomly sampled unseen images.  
The visualizations are detailed in **Section H** of the Appendix. 
The results indicate that while dropout operation can alleviate the co-adaptation phenomenon in the last layer, it fails to address the issue in the shallow layers. 
In contrast, incorporating our proposed training strategy effectively mitigates co-adaptation and achieves equitable channel responses in both shallow and deep layers. 

[1] Kong X, Liu X, Gu J, et al. Reflash dropout in image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 6002-6012.

Reviewer comment (Experiments):
The authors have performed the ablation studies about the ratio of layers involved in EATS, indicating that the increased involvement of layers leads to better generalization. How does the number of filters involved in each training iteration affect the generalization performance?

Author response:
**Ablation studies**

Thank you for your insightful suggestion. 
Ablation studies on the number of filters involved in each iteration are detailed in **Section F** of the Appendix.
Our experiments explored configurations with 1 (default), 3, and 5 filters engaged in each iteration.
Compared to the default of 1 filter, involving 3 filters achieved only a slight improvement. 
This decline may be attributed to increased randomness and instability during the training process.

|         Config         |   Set5   |          |          |          |          |          |          |          |
|:----------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|                        |   clean  |   blur   |   noise  |   jpeg   |    b+n   |    b+j   |    n+j   |   b+n+j  |
|        Original        | 25.2688  | 25.2776  | 22.2421  | 23.9881  | 23.3838  | 23.6890  | 23.4186  | 22.9922  |
| EATS-1filter (default) | 26.0074  | 25.7857  | 22.5926  | 24.4301  | 23.4469  | 23.8805  | 23.6479  | 23.0888  |
|      EATS-3filter      | 26.0538  | 25.9900  | 22.5482  | 24.4395  | 23.6294  | 23.9155  | 23.5702  | 23.0095  |
|      EATS-5filter      | 25.8929  | 25.6534  | 22.4461  | 24.2632  | 23.5130  | 23.8032  | 23.5774  | 23.0124  |
|          Set14         |          |          |          |          |          |          |          |          |
|        Original        | 22.9262  | 22.8528  | 20.9357  | 22.1325  | 22.0795  | 21.9733  | 21.9026  | 21.6333  |
| EATS-1filter (default) | 23.4922  | 23.2615  | 21.2980  | 22.4989  | 22.1894  | 22.1938  | 22.1133  | 21.7674  |
|      EATS-3filter      | 23.6219  | 23.3394  | 21.3617  | 22.5045  | 22.3048  | 22.1650  | 22.0960  | 21.7275  |
|      EATS-5filter      | 23.4872  | 23.1491  | 21.2607  | 22.4014  | 22.0382  | 22.1274  | 22.0476  | 21.7356  |
|         BSDS100        |          |          |          |          |          |          |          |          |
|        Original        | 23.3654  | 23.3880  | 21.2915  | 22.7456  | 22.3154  | 22.6020  | 22.4485  | 22.2281  |
| EATS-1filter (default) | 23.6963  | 23.6864  | 21.5848  | 22.9928  | 22.4043  | 22.7814  | 22.5743  | 22.3039  |
|      EATS-3filter      | 23.7743  | 23.7075  | 21.7250  | 23.0244  | 22.5536  | 22.8058  | 22.5950  | 22.3234  |
|      EATS-5filter      | 23.5557  | 23.5699  | 21.4584  | 22.9784  | 22.5033  | 22.7022  | 22.5667  | 22.3020  |
|        Urban100        |          |          |          |          |          |          |          |          |
|        Original        | 21.5738  | 21.4637  | 19.5775  | 20.9649  | 20.4764  | 20.5970  | 20.7884  | 20.3406  |
| EATS-1filter (default) | 21.9979  | 21.6462  | 19.8111  | 21.2543  | 20.5088  | 20.7175  | 20.9688  | 20.4000  |
|      EATS-3filter      | 21.9795  | 21.7051  | 19.9301  | 21.2136  | 20.6727  | 20.6532  | 20.9733  | 20.3649  |
|      EATS-5filter      | 21.8009  | 21.5756  | 19.7940  | 21.1980  | 20.4379  | 20.6861  | 20.9068  | 20.3656  |
|        Manga100        |          |          |          |          |          |          |          |          |
|        Original        | 18.6101  | 18.8597  | 18.4166  | 18.5256  | 19.0068  | 18.6980  | 18.5759  | 18.6658  |
| EATS-1filter (default) | 19.9220  | 19.0082  | 18.6622  | 18.8107  | 19.1926  | 18.9026  | 18.7213  | 18.7592  |
|      EATS-3filter      | 19.9446  | 19.2506  | 18.8891  | 18.8846  | 19.2921  | 19.9487  | 18.7675  | 18.7990  |
|      EATS-5filter      | 19.8026  | 18.9324  | 18.6103  | 18.8145  | 19.1652  | 18.8039  | 18.7179  | 18.7605  |

Reviewer comment (Evaluation):
The authors are encouraged to provide the computational costs introduced by the proposed training strategy. This addition would help reviewers to assess the practical implications and feasibility of the proposed approach.

Author response:
**Computational costs incurred by our EATS**

The two forward propagations in our training paradigm marginally increase training times, approximately by 1.4 times compared to the corresponding original networks. 
This increment is deemed acceptable, especially considering the significant improvement in generalization capability.
Additionally, it's important to highlight that our training paradigm is exclusively applied during the training process and does not impose any additional computational burden during the inference phase.
We have added the discussion in the **Section C** of Appendix.

Reviewer comment (Experiments):
The ablation studies about the impact of varying layer ratios involved in EATS make me confused. I'm uncertain about how the authors divided the involved layers. Was it through random sampling, division from shallow to deep layers, or perhaps another approach? If the division was from shallow to deep, please explain why more deep layers involved results in better performance.

Author response:
**Clarification about ablation studies about the impact of varying layer ratios involved in EATS**

Thank you for bringing up this concern. The involved layers in our proposed EATS are randomly sampled in advance before the training process. 
Subsequently, the EATS is applied to the selected layers. 
The experimental results consistently demonstrate that involving more layers leads to better generalization performance. This aligns with our objective of awakening a greater number of neurons and promoting equitable contributions across all neurons.

Reviewer comment (Evaluation):
The implementation of EATS involves an awakening regularization term, which emphasizes the contribution of the erased filter by constraining predictions of the disrupted network approximates the baseline image. However, the random sampling of layers and filters in each training iteration has raised concerns about the convergence of the awakening regularization term. Therefore, the authors are encouraged to provide the curve of the regularization in the training process.

Author response:
**The convergence of the awakening regularization**

Thanks for the suggestion. We have incorporated the visualizations of the original loss, $L_{ori}$, and our awakening regularization, $L_{awa}$, during the training process of SRResNet-EATS and RRDBNet-EATS. 
These visualizations can be found in **Section D** of Appendix. 
Through the plots, it is evident that the awakening regularization progressively decreases and converges with the training iteration.

Reviewer comment (Evaluation):
The proposed EATS requires an additional forward propagation step during each training iteration. It is crucial to quantify the extra training time incurred by this strategy compared to the original baseline algorithm.

Author response:
**Training time incurred by our EATS**

The two forward propagations in our training paradigm marginally increase training times, approximately by 1.4 times compared to the corresponding original networks. 
This increment is deemed acceptable, especially considering the significant improvement in generalization capability.
Additionally, it's important to highlight that our training paradigm is exclusively applied during the training process and does not impose any additional computational burden during the inference phase.
We have added the discussion in the **Section C** of Appendix.

Reviewer comment (Experiments):
Figure 1 in the manuscript presents the channel responses of the shallow layer, 2nd block, in SRResNet and SRResNet-EATS. However, it would be valuable for reviewers to also observe if similar phenomena occur in deep layers, where channel responses are awakened and equitable. Visualizations of channel responses from shallow to deep layers would provide a more comprehensive understanding of the awakening effect across various network depths.

Author response:
**Activating channel responses in the deep layer**

We conducted an analysis of channel responses from a deep layer, specifically the output block, in SRResNet (trained on Five5K) on 100 randomly sampled unseen images. 
The visualizations are presented in **Section H** of the Appendix. 
Similar to the observed phenomenon in the shallow layer, the deep layer of the SRResNet exhibits co-adaptation, where a few channels are highly activated while others are inhibited. 
In contrast, SRResNet incorporated with our EATS demonstrates equitable and activated channel responses. 
This indicates that our proposed training strategy is effective in alleviating the co-adaptation and activating channel responses from shallow to deep layers.

Reviewer comment (Experiments):
The experiments could have used more recent SISR models instead of SRResNet and RRDBNet, which have been more than 5 years old. One possible way to alleviate this issue is to replicate Table 1 and Table 2 results in the paper using more recent attention-based architectures like HAN [3], SAN [4], or SwinIR [5] to demonstrate the effectiveness of EATS, if possible.

Author response:
**Evaluation on attention-based algorithm**

Thank you for your valuable suggestion. 
Given the time constraints, we have evaluated the effectiveness of our proposed EATS using a lightweight variant of HAN [1] with x4 scaling. 
This lightweight version comprises approximately 1/20 of the parameters found in the original network.
To maintain consistency with the experimental settings outlined in the manuscript, we have re-trained both the lightweight HAN and HAN-EATS, incorporating high-order degradation modeling. 
The quantitative results are presented below. Additionally, we plan to conduct further experiments on the x2 scale and will include the updated results in the **Section B** of Appendix upon acceptance of this paper.

|    Config   |   Set5   |          |          |          |          |          |          |          |
|:-----------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|             |   clean  |   blur   |   noise  |   jpeg   |    b+n   |    b+j   |    n+j   |   b+n+j  |
|   Original  | 25.4834  | 25.4207  | 22.5964  | 24.1479  | 23.7007  | 23.8615  | 23.5668  | 23.1275  |
|     EATS    | 26.1176  | 25.8898  | 22.8330  | 24.4354  | 23.8923  | 23.9615  | 23.6703  | 23.1699  |
| Improvement |  0.6342  |  0.4691  |  0.2366  |  0.2875  |  0.1916  |  0.1000  |  0.1035  |  0.0424  |
|    Set14    |          |          |          |          |          |          |          |          |
|   Original  | 23.0610  | 23.1144  | 21.3902  | 22.2880  | 22.3199  | 22.2746  | 22.0212  | 21.7860  |
|     EATS    | 23.4764  | 23.3560  | 21.5058  | 22.4861  | 22.3971  | 22.2772  | 22.1089  | 21.7872  |
| Improvement |  0.4154  |  0.2416  |  0.1156  |  0.1981  |  0.0772  |  0.0026  |  0.0877  |  0.0012  |
|   BSDS100   |          |          |          |          |          |          |          |          |
|   Original  | 23.4619  | 23.4352  | 21.7799  | 22.8227  | 22.5801  | 22.7466  | 22.4817  | 22.2781  |
|     EATS    | 23.6903  | 23.5756  | 21.8599  | 22.9232  | 22.6394  | 22.7573  | 22.5339  | 22.2937  |
| Improvement |  0.2284  |  0.1404  |  0.0800  |  0.1005  |  0.0593  |  0.0107  |  0.0522  |  0.0156  |
|   Urban100  |          |          |          |          |          |          |          |          |
|   Original  | 21.6242  | 21.5168  | 20.2169  | 21.0329  | 20.7608  | 20.7810  | 20.7895  | 20.3836  |
|     EATS    | 21.9152  | 21.7400  | 20.2708  | 21.1839  | 20.7677  | 20.8224  | 20.8381  | 20.3987  |
| Improvement |  0.2910  |  0.2232  |  0.0539  |  0.1510  |  0.0069  |  0.0414  |  0.0486  |  0.0151  |
|   Manga100  |          |          |          |          |          |          |          |          |
|   Original  | 18.9412  | 19.2265  | 18.6995  | 18.7634  | 19.1281  | 19.0087  | 18.7372  | 18.7624  |
|     EATS    | 19.3123  | 19.5149  | 18.8154  | 19.0205  | 19.2600  | 19.1412  | 18.8486  | 18.9039  |
| Improvement |  0.3711  |  0.2884  |  0.1159  |  0.2571  |  0.1319  |  0.1325  |  0.1114  |  0.1415  |

[1] Niu, Ben, et al. "Single image super-resolution via a holistic attention network." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16. Springer International Publishing, 2020. https://arxiv.org/abs/2008.08767

Reviewer comment (Writing):
Minor typos exist in the text. For instance, 'Shapely' should be 'Shapley' in the abstract and the Solution subpoint on the 2nd page. In addition to that, 'Managa109' should be 'Manga109' [6] on Table 2 on the 9th page.

Author response:
**Typos**

Thank you for bringing these typos to our attention. We appreciate your keen observation. The corrections have been addressed in the revision.

Reviewer comment (Experiments):
To my understanding, since the model uses two networks: original network $f_{\theta}$ and disrupted network $f_{\theta'}$, does that mean that the whole framework involves more parameters (approximately doubles) to that of the baselines? If yes, can authors address the # of parameters involved in employing said method with EATS?

Author response:
**Clarification regarding the parameters involved in our training paradigm**  

Firstly, it's essential to emphasize that our proposed Erasing and Awakening Training Scheme (EATS) does not introduce additional training parameters.
Within a training iteration, we conduct a forward propagation of the original network, $f_θ$, calculating the original loss function, $L_{ori}$. 
Subsequently, our EATS involves randomly replacing an erased filter with a disrupted filter based on the original network, resulting in a modified network after erasure, $f_{θ^{'}}$. 
We then calculate the awakening regularization, $L_{awa}$, through a second forward propagation. 
The network is update based on the combination of these two loss functions.
Therefore, it's crucial to note that our EATS only requires two forward propagations, as opposed to employing two separate neural networks. 

Secondly, the two forward propagations in our training paradigm marginally increase training times, approximately by 1.4 times compared to the corresponding original networks. 
This increment is deemed acceptable, especially considering the significant improvement in generalization capability.

Lastly, it's important to highlight that our training paradigm is exclusively applied during the training process and does not impose any additional computational burden during the inference phase.

We have added the discussion in the **Section C** of Appendix.