Paper: Lightweight Image Super-Resolution via Flexible Meta Pruning (ICLR-2024)

Reviewer comment (Writing):
The introduction of the proposed methods appears somewhat confusing. The proposed pruning method combines both structured and unstructured pruning, but it is difficult to distinguish their specific implementations and how they are integrated when reading this paper. Are channels being pruned and weights set to zero, or is it only utilizing sparse regularization for structural pruning.

Author response:
Thank you for your valuable feedback. 

**(1)** Specific implementations. We optimize the channel vectors and weight indicators with proximal gradient and SGD. There are mainly two reasons: **First**, SGD with regularization could already lead to acceptable weight sparsity while the channel sparsity could be harder to achieve. Thus, an advanced algorithm proximal gradient descent (PGD) is used for channel pruning to accelerate the convergence of the optimization procedure. **Second**, by separating the optimization, we are more flexible in controlling the contribution of channel pruning and weight pruning.

**(2)** How they are integrated? During the optimization, we set regularization factors $\lambda_C$=0.1 and $\lambda_W$=0.02 for both structured pruning and unstructured pruning. If the compression ratio of one pruning method is achieved, the optimization method for either the channel vectors or the weight indicators is stopped while the other one continues. The whole algorithm converges if both of the compression targets are achieved.

**(3)** Network pruning output. The pruned channels are removed. The pruned weights are set to zero. We conduct both structural and unstructural pruning.

We will revise the paper and make it clearer based on your comments.

Reviewer comment (Evaluation):
The comparison with existing methods lacks a comparison of runtime.

Author response:
Thanks for the valuable suggestion. We provide inference time with other methods in the following Table. EDSR-16-256 means 16 residual blocks (RBs) with 256 channels for each Conv layer in RB. The inference time is tested on Urban100 ($\times$4) with an RTX 3090.

   |  Method   | EDSR-16-256 |   CARN   |   IMDN   |  DHP   |  ASSLN |  FMP (ours) |
   |  -------  |  :-------:  | :------: | :------: | :----: | :----: | :----:      |
   | Time (ms) |     1397    |    53    |     39   |   49   |  49    |   21        |

According to the inference time comparison, we can see that our FMP achieves less inference time than others.

Reviewer comment (Presentation):
Table 6 is challenging to understand.

Author response:
Thanks for the valuable comment. 

**(1)** Table 6 is about the results of different convergence criteria. We provided explanations about convergence criteria in Sec. 4.4 of the main paper. 

**(2)** In Sec. 4.4, we provided a brief analysis about Table 6. Meanwhile, in Sec. 2.2 of the supplementary material, we provided further analyses about convergence criteria.

Based on the reviewer's comments, we will add more content from supplementary material to the main paper, trying to make Table 6 easier to understand.

Reviewer comment (Novelty):
The advantages of this method in terms of flexibility compared to standalone structured and unstructured pruning need to be further emphasized.

Author response:
Thanks for the valuable suggestion. 

**(1)** In Sec. 3.1 of the main paper, we give some reasons why we need flexible pruning. This is a part of our motivation, where we also briefly discuss the advantages of our FMP over standalone structured and unstructured pruning.

**(2)** In Sec. 1.1 of the supplementary material, we provide more clarifications and explanations about the motivation of our method.

We will revise Sec. 3.1 of the main paper to further emphasize our advantages as suggested by the reviewer.

Reviewer comment (Reproducibility):
The proposed pruning method combines both structured and unstructured pruning, but it is difficult to distinguish their specific implementations and how they are integrated when reading this paper. Are channels being pruned and weights set to zero, or is it only utilizing sparse regularization for structural pruning.

Author response:
Thanks for the valuable question.

**(1)** Specific implementations. We optimize the channel vectors and weight indicators with proximal gradient and SGD. There are mainly two reasons: **First**, SGD with regularization could already lead to acceptable weight sparsity while the channel sparsity could be harder to achieve. Thus, an advanced algorithm proximal gradient descent (PGD) is used for channel pruning to accelerate the convergence of the optimization procedure. **Second**, by separating the optimization, we are more flexible in controlling the contribution of channel pruning and weight pruning.

**(2)** How they are integrated? During the optimization, we set regularization factors $\lambda_C$=0.1 and $\lambda_W$=0.02 for both structured pruning and unstructured pruning. If the compression ratio of one pruning method is achieved, the optimization method for either the channel vectors or the weight indicators is stopped while the other one continues. The whole algorithm converges if both of the compression targets are achieved.

**(3)** Network pruning output. The pruned channels are removed. The pruned weights are set to zero. We conduct both structural and unstructural pruning.

We will revise the paper and make it clearer based on your comments.

Reviewer comment (Experiments):
As a pruning method, how does it perform when applied to existing super-resolution networks?

Author response:
Thanks for the valuable question about the generalization ability of our method.

**(1)** In this work, we mainly apply our FMP to CNN-based EDSR and newly designed LSRB. To further demonstrate the generalization ability of the network pruning methods, we should also have applied FMP to several classic image SR models. Although LSRB is closely related to several leading lightweight networks (e.g., IMDN, RFAN, RLFN), it is more convincing to try more SR models. 

**(2)** It is worth investigating FMP in Transformer-based methods (e.g., SwinIR).

**(3)** Our FMP is a general idea for flexible structured and unsructred pruning. We can apply FMP to CNN-based and Transformer-based image SR methods.

We will further investigate such a direction and provide more baselines.

Reviewer comment (Novelty):
The manuscript's contribution seems to be incremental, primarily involving the addition of kernel weight pruning based on DHP [1]. While the primary distinction between the proposed method and DHP lies in kernel pruning, the results shown in Table 6 indicate that simultaneous channel and kernel pruning may not significantly show effectiveness over channel pruning alone.

Author response:
Thank you for your valuable feedback.

**(1)** In the first sentence of Sec. 4.3 of main paper, we state that training process will stop if meets convergence or reach the maximum iterations 300K. We find that our FMP would meet convergence with fewer iterations than DHP. But, this is mainly based on the validation performance to early stop the training. 

**(2)** When we stop training our FMP with fewer iterations than DHP, our model may not achieve obvious improvements than DHP on other datasets, like Urban100. 

**(3)** We allow early stop for the ablation study is mainly to save the training time with limited resource. We hope to investigate if it is possible for us to train the lightweight models with fewer iterations, which we think is also favorable in real practice.

Of course, we will revise the paper and make this part clearer based on the reviewer's comments.

Reviewer comment (Experiments):
Some of the experimental comparisons may be perceived as lacking in both fairness and completeness when it comes to demonstrating the effectiveness of the method. Notably, in Figure 1, the analysis exclusively focuses on showcasing the influence of FMP. However, it's important to note that the baseline models utilized for evaluating FMP and other comparative methods differ. This dissimilarity in the baseline models pose challenges when attempting to assess the precise impact of FMP. Besides, in subsection 4.2, the experimental comparisons fail to cover some classic lightweight SR models including RLFN and ELAN, which are also mentioned in Section 1 and Section 2.

Author response:
Thanks for the valuable suggestions. 

**(1)** In Figure 1, we mainly want to give an overall view of our method FMP. Specifically, we show that an image SR baseline with FMP can achieve comparable or better results than recent methods.

**(2)** To showcase the influence of FMP, we main provide corresponding results in ablation part.

**(3)** We compare with more recent works, like SRPN [Ref1], RLFN [Ref2], and ELAN [Ref3]. We provide PSNR/SSIM values of compared methods in the following table. 

   |  Method  |   Scale    |    Set5        |   Set14        |   B100         |  Urban100      |   
   |  ------- | :-------:  | :-------:      | :------:       | :------:       | :----:         |
   |   SRPN   | $\times$2  | 38.10 / 0.9608 | 33.70 / 0.9189 | 32.25 / 0.9005 | 32.26 / 0.9294 |  
   |   RLFN   | $\times$2  | 38.07 / 0.9607 | 33.72 / 0.9187 | 32.22 / 0.9000 | 32.33 / 0.9299 |   
   |   ELAN   | $\times$2  | 38.17 / 0.9611 | 33.94 / 0.9207 | 32.30 / 0.9012 | 32.76 / 0.9340 |  
   |FMP (ours)| $\times$2  | 38.17 / 0.9615 | 33.81 / 0.9215 | 32.32 / 0.9022 | 32.71 / 0.9360 |  
   |   SRPN   | $\times$3  | 34.47 / 0.9276 | 30.38 / 0.8425 | 29.16 / 0.8061 | 28.22 / 0.8534 |  
   |   RLFN   | $\times$3  | N/A            | N/A            | N/A            | N/A            |  
   |   ELAN   | $\times$3  | 34.61 / 0.9288 | 30.55 / 0.8463 | 29.21 / 0.8081 | 28.69 / 0.8624 |  
   |FMP (ours)| $\times$3  | 34.55 / 0.9291 | 30.48 / 0.8456 | 29.20 / 0.8101 | 28.40 / 0.8597 |  
   |   SRPN   | $\times$4  | 32.24 / 0.8958 | 28.69 / 0.7836 | 27.63 / 0.7373 | 26.16 / 0.7875 |  
   |   RLFN   | $\times$4  | 32.24 / 0.8952 | 28.62 / 0.7813 | 27.60 / 0.7364 | 26.17 / 0.7877 |  
   |   ELAN   | $\times$4  | 32.43 / 0.8975 | 28.78 / 0.7858 | 27.69 / 0.7406 | 26.54 / 0.7982 |
   |FMP (ours)| $\times$4  | 32.34 / 0.8979 | 28.71 / 0.7878 | 27.67 / 0.7425 | 26.35 / 0.7954 |  

According to the table, we can see that our FMP achieves comparable performance than recent leading methods.

We will revise the paper with more recent methods based on the suggestios.

[Ref1] Learning Efficient Image Super-Resolution Networks via Structure-Regularized Pruning, ICLR, 2022

[Ref2] Residual Local Feature Network for Efficient Super-Resolution, CVPRW, 2022

[Ref3] Efficient Long-range Attention Network for Image Super-Resolution, ECCV, 2022

Reviewer comment (Experiments):
The manuscript does not sufficiently address the compatibility and generalizability of the proposed FMP method. Although FMP was developed for lightweight super-resolution, experiments have only been conducted on one model, LSRB. The suitability of applying FMP to other super-resolution models remains unexplained.

Author response:
Thanks for the valuable suggestion.

**(1)** In this work, we mainly apply our FMP to CNN-based EDSR and newly designed LSRB. To further demonstrate the generalization ability of the network pruning methods, we should also have applied FMP to several classic image SR models. Although LSRB is closely related to several leading lightweight networks (e.g., IMDN, RFAN, RLFN), it is more convincing to try more SR models. We can mainly replace the LSRB with other SR methods and use hypernetwork to generate their corresponding parameters.

**(2)** It is worth investigating FMP in Transformer-based methods (e.g., SwinIR). 

**(3)** Our FMP is a general idea for flexible structured and unsructred pruning. We can apply FMP to CNN-based and Transformer-based image SR methods.

We will further investigate such a direction and provide more baselines.

Reviewer comment (Reproducibility):
What optimization method was employed for the FMP method outlined in the manuscript? While the introduction section mentions SGD and proximal gradient descent, the experimental setup section suggests the use of Adam for FMP. This inconsistency is expected to be clarified.

Author response:
Thanks for pointing this part out.

**(1)** We optimize the channel vectors and weight indicators with proximal gradient and SGD. There are mainly two reasons: **First**, SGD with regularization could already lead to acceptable weight sparsity while the channel sparsity could be harder to achieve. Thus, an advanced algorithm proximal gradient descent (PGD) is used for channel pruning to accelerate the convergence of the optimization procedure. **Second**, by separating the optimization, we are more flexible in controlling the contribution of channel pruning and weight pruning.

**(2)** When we train the image SR network with flexible meta pruning jointly, we have to update the **whole network parameters**. In this procesure, we use the general ADAM optimizer, which is also adopted in most of other image SR methods.

We will further clarify them in the paper based on the suggestion.

Reviewer comment (Evaluation):
LSFB represents a lightweight network achieved by replacing residual blocks in RLFN with simplified versions, and FMP is a lightweight method. Table 3 indicates that LSFB+FMP exhibits higher model parameters and computational complexity compared to RLFN. Could you provide an explanation for this discrepancy? Furthermore, LSFB+FMP demonstrates increased inference speed, despite the primary goal of parameter optimization not being fast adaptation in meta-learning. The reasons behind the improved inference speed is expected to be addressed and analyzed.

Author response:
Thanks for the valuable suggestion. 

In Table 3 of the main paper, our LSRB has slightly larger number of parameters and FLOPs, but achieves faster speed. The reasons are mainly summarized as follows:

**(1)** In the building block RLRB in RLFN, there are three convolutional layers (each layer is followed by ReLU) with channel number set as 52. While, the building block in our LSRB-6-48 has two convolutional layers (only the first convolutional layer is followed by ReLU) with channel number as 48.

**(2)** Currently, in PyTorch, as used in our work, networks with wider width (i.e., larger channel number) will run slower than narrower one with similar model size. So, LSRB runs faster than RLFN.

**(3)** As we state in Effectiveness of LSRB in Sec. 4.3, it is promising to further reduce its redundant parameters with our proposed FMP. In our research, we can hardly use our common GPU with PyTorch to achieve further acceleration. But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning.

We will include those clarifications and explanations in the revised paper.

Reviewer comment (Evaluation):
FMP introduces kernel weight pruning in addition to DHP, and the results in Table 4 is intended to show that FMP outperformances DHP. However, the combination of multiple pruning techniques may not always lead to an improved result. Besides, it seems that the higher the prune ratio, the better for the performances of both DHP and FMP. Both appear to be counterintuitive to some extents. Therefore, more detailed analysis and explanation are necessary here.

Author response:
Thanks for the valuable comments and suggestions. 

**(1)** In the first sentence of Sec. 4.3 of main paper, we state that the training process will stop if meets convergence or reach the maximum iterations 300K. We find that our FMP would meet convergence with fewer iterations than DHP. But, this is mainly based on the validation performance to early stop the training. 

**(2)** When we stop training our FMP with fewer iterations than DHP, our model may not achieve obvious improvements than DHP on other datasets, like Urban100. 

**(3)** We allow early stop for the ablation study is mainly to save the training time with limited resource. We hope to investigate if it is possible for us to train the lightweight models with fewer iterations, which we think is also favorable in real practice.

**(4)** When the prune ratio becomes higher, the training needs more iterations. Models with more training iterations would actually get higher performance. 

We will revise the paper with more detailed analyses and explanations.

Reviewer comment (Experiments):
Formula (6) is similar to formula (7) because it needs to calculate the norm as well. However, why not trying to choose the suitable norm through experiments in the same way adopted to determine the norm in formula (7)?

Author response:
Thanks for the interesting question. The main reasons are as follows:

**(1)** Formula (6) is for channel pruning, which has been widely investigated before in image SR, like DHP and ASSL. In previous work, we find that $L_1$ norm is a good choice for channel pruning.

**(2)** We choose $L_1$ norm for channel pruning so that we can keep the same as previous channel pruning works.

**(3)** Unstructured pruning is seldom investigated in image SR before. So, we try different choices of norm and determine the final one for our experiments.

So, in our work, we mainly investigate the choices of norm for Eq. (7), namely unstructured pruning.

Reviewer comment (Evaluation):
The authors argue that unstructured pruning does not contribute to actual acceleration. However, the experiments in Table 1 do not include comparisons of latency.

Author response:
Thank you for your valuable feedback.

**(1)** In Table 1 of the main paper, it is not easy to provide inference time for all the compared methods. Many of them are officially implemented with different frameworks. For example, SRCNN, FSRCNN, VDSR, DRCN, and MemNet are officially implemented with Caffe. LapSRN is officially implemented with MatConvNet. To compare their inference time fairly, we should implement them under the same codebase and framework and run the models with the same GPU. Currently, we do not have such a codebase. But, it is important to have such a benchmark codebase in the future.

**(2)** Instead, we provide inference time with other methods in the following Table. EDSR-16-256 means 16 residual blocks (RBs) with 256 channels for each Conv layer in RB. The inference time is tested on Urban100 ($\times$4) with an RTX 3090. 

   |  Method   | EDSR-16-256 |   CARN   |   IMDN   |  DHP   |  ASSLN |  FMP (ours) |
   |  -------  |  :-------:  | :------: | :------: | :----: | :----: | :----:      |
   | Time (ms) |     1397    |    53    |     39   |   49   |  49    |   21        |

According to the inference time comparison, we can see that our FMP achieves less inference time than others.

Reviewer comment (Experiments):
Some leading lightweight methods are missed in Table 1, e.g. MegSR[1], VapSR[2], Omni-SR[3]. The authors are suggested to provide comparisons over varied model sizes to verify the effectiveness of the proposed method, because we can not justify whether it can maintain its advantage among smaller models (~400 K params).

Author response:
Thanks for the valuable suggestions. 

We compare with DIPNet[1], VapSR[2], and Omni-SR[3]. PSNR/SSIM values are provided in the following table. It should be noted that we cite the results of DIPNet, VapSR, and Omni-SR from their papers.

   |  Method             |   Scale    | Params (K)|   Set5  |   Set14        |   B100         |  Urban100      |   
   |  -------            | :-------:  |  :-------:|:-------:| :------:       | :------:       | :----:         |
   |  DIPNet (CVPRW'23)  | $\times$2  | 527 |37.98 / 0.9605 | 33.66 / 0.9192 | 32.20 / 0.9002 | 32.31 / 0.9302 |  
   |   VapSR (ECCVW'22)  | $\times$2  | 329 |38.08 / 0.9612 | 33.77 / 0.9195 | 32.27 / 0.9011 | 32.45 / 0.9316 |   
   | Omni-SR (CVPR'23)   | $\times$2  | 772 |38.29 / 0.9617 | 34.27 / 0.9238 | 32.41 / 0.9026 | 33.30 / 0.9386 |  
   |   FMP (ours)        | $\times$2  | 694 |38.17 / 0.9615 | 33.81 / 0.9215 | 32.32 / 0.9022 | 32.71 / 0.9360 | 
   |  DIPNet (CVPRW'23)  | $\times$3  | N/A |N/A            | N/A            | N/A            | N/A            |  
   |   VapSR (ECCVW'22)  | $\times$3  | 337 |34.52 / 0.9284 | 30.53 / 0.8452 | 29.19 / 0.8077 | 28.43 / 0.8583 |   
   | Omni-SR (CVPR'23)   | $\times$3  | 780 |34.77 / 0.9304 | 30.70 / 0.8489 | 29.33 / 0.8111 | 29.12 / 0.8712 |  
   |   FMP (ours)        | $\times$3  | 684 |34.55 / 0.9291 | 30.48 / 0.8456 | 29.20 / 0.8101 | 28.40 / 0.8597 | 
   |  DIPNet (CVPRW'23)  | $\times$4  | 543 |32.20 / 0.8950 | 28.58 / 0.7811 | 27.59 / 0.7364 | 26.16 / 0.7879 |  
   |   VapSR (ECCVW'22)  | $\times$4  | 342 |32.38 / 0.8966 | 28.77 / 0.7852 | 27.68 / 0.7398 | 26.35 / 0.7941 |   
   | Omni-SR (CVPR'23)   | $\times$4  | 792 |32.57 / 0.8993 | 28.95 / 0.7898 | 27.81 / 0.7439 | 26.95 / 0.8105 |  
   |  FMP (ours)         | $\times$4  | 704 |32.34 / 0.8979 | 28.71 / 0.7878 | 27.67 / 0.7425 | 26.35 / 0.7954 | 

According to the above table, we have the following analyses.

**(1)** All the compared methods (i.e., DIPNet, VapSR, Omni-SR) have new network designs. Those new designs can achieve obviously better performance than our ResNet based backbone LSRB.  

**(2)** Network parameter number matters very much. Ommni-SR achieves the best performance with the largest model size.

**(3)** In our work, we mainly focus on baselines with high speed, where we design new models based on the champion solution RLFN at NTIRE'22 Efficient SR. Our FMP can also be applied to new SOTA lightweight image SR methods shown above. It is important and promising to investigate applying our FMP to the above methods in the future.

As suggested by the reviewer, we will further investigate smaller models by pruning more channels and weights.

Reviewer comment (Experiments):
According to Table 3, the backbone used by FMP already brings some improvements. Thus comparisons in Table 7 may be not fair. The authors are suggested to provide the results of applying ASSLN to LSRB.

Author response:
Thanks for the valuable suggestions.

We use the LSRB as the backbone and apply ASSLN to it. As time and GPU resource is limited, we only train the SR model with scale $\times$2. We provide PSNR/SSIM values in the following table.

   |  Method  |   Scale    |    Set5        |   Set14        |   B100         |  Urban100      |   
   |  ------- | :-------:  | :-------:      | :------:       | :------:       | :----:         |
   |   ASSLN  | $\times$2  | 38.12 / 0.9608 | 33.77 / 0.9194 | 32.27 / 0.9007 | 32.41 / 0.9309 |  
   |ASSLN+LSRB| $\times$2  | 38.13 / 0.9611 | 33.79 / 0.9198 | 32.28 / 0.9010 | 32.52 / 0.9320 | 
   |    FMP   | $\times$2  | 38.17 / 0.9615 | 33.81 / 0.9215 | 32.32 / 0.9022 | 32.71 / 0.9360 | 

According to the table, we find that ASSLN+LSRB achieves improvement over the original ASSLN. This observation shows the effectiveness of LSRB. Our FMP still obtains better performance than ASSLN+LSRB. It indicates that flexible pruning (i.e., structured pruning and unstructured pruning) could better reduce network redundancy than channel pruning in image SR.

It should also be noted that ASSLN has to use a large pretrained LSRB model. In contrast, our FMP does not need a pretrained model.

Reviewer comment (Evaluation):
It's unclear how the compressed model would translate to the practical inference speed up, considering there are unstructured weight pruning.

Author response:
Thank you for your valuable feedback. 

**(1)** Our designed lightweight image SR baseline LSRB reduces inference time than the champion solution RLFN at NTIRE'22 Efficient SR challenge. As shown in Table 3 of the main paper, our LSRB achieves much faster than RLFN. 

**(2)** In our work, we can hardly use our common GPU with PyTorch to achieve further acceleration for unstructured pruning. But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning.

**(3)** We apply FMP to LSRB. With PyTorh and common GPU, the speed up of FMP mainly comes from LSRB and channel pruning. As a result, our FMP can still achieve relative speed up.

Reviewer comment (Experiments):
What's the benefits of combining channeling pruning with unstructured weight pruning?

Author response:
Thanks for the question.

**(1)** Structured pruning and unstructured pruning are two important network compression methods that can cut down the model complexity of deep neural networks significantly.

**(2)** As we state in Effectiveness of LSRB in Sec. 4.3, it is promising to further reduce its redundant parameters with our proposed FMP. In our current research, we can hardly use our common GPU with PyTorch to achieve further acceleration. 

**(3)** But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning. As far as we know, there are some specific hardware designs and implementations for unstructured pruning in some institutions, where the inference time can further be reduced. However, we have no access to such hardwares right now. 

**(4)** Instead, we provide the potential of further reducing the network redundancy with both structured and unstructured pruning.

Reviewer comment (Evaluation):
In Table 3, instead of just the submodule, is it possible to show the inference speed up of the entire model?

Author response:
Thanks for the valuable suggestion.

We further provide inference time with other methods in the following Table. EDSR-16-256 means 16 residual blocks (RBs) with 256 channels for each Conv layer in RB. The inference time is tested on Urban100 ($\times$4) with an RTX 3090. 

   |  Method   | EDSR-16-256 |   CARN   |   IMDN   |  DHP   |  ASSLN |  FMP (ours) |
   |  -------  |  :-------:  | :------: | :------: | :----: | :----: | :----:      |
   | Time (ms) |     1397    |    53    |     39   |   49   |  49    |   21        |

According to the inference time comparison, we can see that our FMP achieves less inference time than others.

Reviewer comment (Theory):
What's the motivation and benefits of applying proximal gradient for channels vectors and weight indicators?

Author response:
Thanks for the interesting question.

**(1)** The general idea of this work is first introduced before elaborating on details of our flexible network pruning method. Structured pruning and unstructured pruning are two important network compression methods that can cut down the model complexity of deep neural networks significantly.

**(2)** Structured pruning leaves regular kernels after pruning, which is beneficial for the actual acceleration of the network.

**(3)** Unstructured pruning removes single weights in a kernel and can compress the network without sacrificing too much accuracy of the network.

**(4)** Since the channel vectors and the weight indicator are not intertwined by non-linear operations, we can decouple their optimization for network pruning by different methods.

Reviewer comment (Experiments):
How much performance gain can FMP harvest if using self-ensemble?

Author response:
Thanks for pointing this out. Self-ensemle in the test phase is to rotate and flip input, get different output, and then average them to get the final result.

**(1)** Self-ensemble can further achieve about 0.1-0.2 dB in PSNR for different scales and test dataset.

**(2)** However, self-ensemble would use several times of computation cost. For example, in image SR, one common practice of self-ensemble takes 8 times computational cost as the norm one without self-ensemble. So, in our paper, we do not use this strategy for further performance improvement.