Paper: AdaSR: Adaptive Super Resolution for Cross Platform and Dynamic Runtime Environments (ICLR-2024)

Reviewer comment (Reproducibility):
The training process of AdaSR appears to be complex, making it more challenging to extend to other platforms compared to designing a bespoke SR model for the target platform.

Author response:
The main advantage of this training method is that it allows us to train a single model to be capable of performing inference at various latencies without significant drop in performance. To our knowledge, no other work attempts to build such a model. Our motivation is slightly different in that instead of targeting a specific device with a model, we train a single model that can be used on any targeted devices within a certain hardware scope. We will clarify this in the paper.

Reviewer comment (Experiments):
Why regularization for mapping layer is enough to keep the inference performance when removing the mapping layer? In my experience SR model is very sensitive to that kind of operation. Is there a related analysis or experiment?

Author response:
Thank you for pointing this out. Correct, we also found the SR model is sensitive to changes in removal of this layer. We found that using regularization to make the mapping layer to have very small weights somewhat mitigates this issue. We still observe a drop in performance compared to having the mapping layer. However, having no regularization or too large regularization thresholds can result in a much more significant drop in performance. The threshold we used has been tuned to make the performance drop to what we feel are within acceptable limits. We will add these details in the paper.

Reviewer comment (Reproducibility):
This approach requires retraining of existing models. What are the associated training costs in terms of time and hardware expenses?

Author response:
Since we perform layer-wise training, we undergo more back-propagation iterations than if we were to train the full model at once. The total number of iterations would be *N* x *C* x *e* where *N* is the number of blocks, *C* is the number of channel sizes. and *e* the number of epochs till convergence. However, the relative cost of each backpropagation is cheaper since we freeze all previous layers, and only train one layer at a time. To give an estimate, our base CARN model requires 8 GPU days to fully train to convergence on a 1080Ti GPU. We will add these details to our paper.

Reviewer comment (Reproducibility):
In Chapter 3.2.1, it is mentioned, “... increase the size of the adaptable model, ... and repeat the process ...”. The question is: When increasing the size each time, are all blocks in the adaptable model synchronized? For instance, do block 1 through block M increase to the same block size and channel size each time? If this is the case, does this approach iterate through all possible solutions within the search space?

Author response:
Yes, we iterate through each possible solution. However, we can reduce this exploration space depending on what devices we may possibly target. We will add this detail to the paper.

Reviewer comment (Experiments):
In the experimental setup, the parameters beta_1 and beta_2 for the ADAM optimizer are set to the same values as those in the FAKD approach. Are there specific reasons for maintaining these parameters consistent with FAKD?

Author response:
We tried several other parameters, and this gave the overall best results. That said, this had little performance impact compared to others.

Reviewer comment (Experiments):
It would be beneficial to include information about the available memory spaces in the experiments, as it is a crucial factor in characterizing the dynamic runtime environment.

Author response:
This is a good point, we will add this in our results.

Reviewer comment (Presentation):
In Table 1, the significance of the bold values is ambiguous. They do not represent the best performance in the comparisons. More detailed clarifications are required.

Author response:
Thank you for pointing this out. We will remove the bold font from our model names.

Reviewer comment (Experiments):
Chapter 4.2 does not sufficiently demonstrate AdaSR's robust adaptability in a dynamic runtime environment. An ablation study may be necessary, as it is an important statement declared in the Introduction. This is particularly relevant for showcasing AdaSR's adaptability in a dynamic runtime when “other running applications are present.”

Author response:
Thank you for pointing this out. We will add more ablation studies on this.

Reviewer comment (Writing):
A minor writing issue: In Chapter 3.3, there is a phrase, “... we a training method with ...”

Author response:
Thank you, we will fix this.

Reviewer comment (Novelty):
The novelty of this paper is quite limited. It seems that AdaSR adapts the width and depth. It's pretty similar to many previous works such as [*]. The only novelty I could find in AdaSR is to combine progressive distillation and NAS together.

Author response:
Our main novelty is that we can do this in a single model, without having to create and train a new model for every different platform. We will clarify this in the paper.

Reviewer comment (Experiments):
The performance improvement is marginal compared with MDDC. It's hard to tell which method is better based on Table 1. I suggest the author provide a more comprehensive experiment report. Also, the authors should compare with more recent baselines. By the way, I couldn't find any citations in your paper about MDDC in Table 1.

Author response:
The purpose of our paper is not to beat state-of-the-art, but to achieve comparative performance while being able to dynamically compress a single model according to a device’s need. We will clarify this in the paper.

Reviewer comment (Evaluation):
Lack of experiment results on perceptual metrics. I suggest the authors report LPIPS results.

Author response:
Thank you for pointing this out. We will add them to the Appendix

Reviewer comment (Presentation):
The authors use tons of indentation between figures, paragraphs and headings. I'm not sure if this kind of behavior violate ICLR instructions.

Author response:
Thank you for clarifying this, we will re-format where required.

Reviewer comment (Experiments):
As claimed in the paper, "none ... address the challenges in dynamic runtime environment ..." and the proposed AdaSR aims to address this issue. It is confusing and ambiguous for the experiment settings for the cross-platform and dynamic runtime environment.

Author response:
Thank you for this feedback. Our claim of being suitable for dynamic runtime environments hinges on the fact that our model can change its inference latency on the fly without requiring the reloading of models from disk to memory. We will clarify this in the paper. For our experiments for cross-platform compatibility, we demonstrate that a single model can perform at various latencies across various platforms without the need to reload new models (see Figure 4). This is not achieved by the state-of-the-art we compare against. For demonstration of dynamic runtime effectiveness, in Figure 4 that our model can achieve a variety of latencies in the same session dynamically. We will work on making these points clearer in the paper.

Reviewer comment (Experiments):
The evaluations for "cross-platform Pareto optimality" (Sec. 4.1) and "dynamic runtime environment" (Sec. 4.2) are conducted on a very small test set, i.e., set14, Fig.3 and Fig.4. I do not think those results are convincing.

Author response:
We perform our experiments on a variety of test sets, as shown in Table 1. We limit to plotting only the graphs in Figure 3 due to space. We will add the rest of the experimental results in the Appendix.

Reviewer comment (Evaluation):
In Sec. 4.3, "to support adapting models for cross-platform deployment and dynamic runtime environment, AdaSR achieves state-of-the-art performance, AdaSR achieves state-of-the-art performance". However, Tab.1 just shows the results on existing SR datasets. How can we learn from those evaluation results and how to demonstrate the effectiveness under cross-platform deployment and dynamic runtime environment?

Author response:
This section is for directly comparing model performance to similarly sized models. We show here that our method can effectively reduce the model sizes with minimal loss of performance since it is competitive with other state-of-the-art.