Paper: Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR-2024)

Reviewer comment (Experiments):
There are micro, mini and extended versions of the Meta-Album datasets, which poses the question if Quick-Tune could benefit from using the smaller subsets for more quickly finding settings that work well on the larger subsets. It would be interesting to study this to see if it works for Quick-Tune and how large speedups can be obtained.

Author response:
We thank the reviewer for raising an interesting point. We would like to mention that this idea has already been investigated and verified by previous work [1]. As the number of epochs is the most common fidelity used in the domain, we focused on that. However, the fidelity in itself is a proxy and it could as well be dataset size.

[1] Klein, A., Falkner, S., Bartels, S., Hennig, P., & Hutter, F. (2017, April). Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics (pp. 528-536). PMLR.

Reviewer comment (Experiments):
How well does the method work on other downstream tasks or data modalities?

Author response:
The reason that our method is focused on computer vision classification is that there already exists a well-established model hub in computer vision that is used by the community. However, we see no reason why our method would not work on other data modalities. We agree with the reviewer that it is infeasible to run on other data modalities at this stage and we believe this is part of future work.

Reviewer comment (Experiments):
The reviewer is curious about how the method performs on other standard benchmarks such as Imagenet.

Author response:
Dear reviewer,

thank you for the feedback. We would like to clarify that the models considered in our search space, have been pretrained in ImageNet, which is why we do not include exactly ImageNet in our experiments. However, we would like to point the reviewer to Section C of the appendix, where we have done exactly what the reviewer suggests and we have evaluated how our method performs on other standard benchmarks such as Imagenette2-320 (a smaller version of ImageNet) and iNaturalist [2] against the baselines.

[1]https://www.tensorflow.org/datasets/catalog/imagenette

[2] Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., ... & Belongie, S. (2018). The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8769-8778).

Reviewer comment (Experiments):
The reviewer is curious whether any of these results can be validated on different domains (e.g., text datasets, where finetuning is also very common).

Author response:
Dear reviewer,

thank you for the feedback. We would like to clarify that the models considered in our search space, have been pretrained in ImageNet, which is why we do not include exactly ImageNet in our experiments. However, we would like to point the reviewer to Section C of the appendix, where we have done exactly what the reviewer suggests and we have evaluated how our method performs on other standard benchmarks such as Imagenette2-320 (a smaller version of ImageNet) and iNaturalist [2] against the baselines.

We agree with the reviewer and we believe that our work opens up the chance to research further applications and model hubs. For instance, there is currently an explosion of new model hubs for LLMs in the community, such as OpenLLM, and we are looking forward to applying our methodology to this domain in future work.

[1]https://www.tensorflow.org/datasets/catalog/imagenette

[2] Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., ... & Belongie, S. (2018). The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8769-8778).

Reviewer comment (Experiments):
In the ablation study, for the Micro and Mini cases, QT: -M, +C, +G (DyHPO with cost-awareness) performs as well as QT: +M, +C, +G (DyHPO with cost-awareness and meta-learning) which shows most of the benefit coming from the cost-aware aspect of QuickTune and not the meta-learning.

Author response:
We thank the reviewer for raising an interesting point. We believe that Micro and Mini feature small tasks and as such, the time overhead of the initial evaluations does not penalize the performance during the optimization as much. In the case of more expensive tasks, this effect is reversed, and the meta-learning aspect helps improve performance by making use of prior information. We believe that meta-learning is an important aspect of our method, and in the case of small tasks it achieves similar performance to the non-meta-learned version.

Reviewer comment (Novelty):
Novelty is limited since the core of Quick-Tune is DyHPO, a prior HPO method. Cost-aware acquisition functions have been used in the past and the approach of using meta-features and meta-learning good initializations for the estimators also lack originality.

Author response:
Although our method extends previous work on gray-box optimization, we want to highlight that we make several novel contributions: 1) we show how to effectively combine meta-learning and cost-awareness in a gray-box setup, 2) we design a search space for finetuning based on a large model hub,  3) we introduce a meta-dataset for analysis and meta-learning, and 4) we show how to extend these ideas into the automated model selection and hyperparameter optimization for finetuning pretrained models.

Reviewer comment (Evaluation):
What is the unnormalized performance of the approaches in Figure 3 and Figure 4?

Author response:
We use the normalized performance since it makes the performance aggregation between the tasks correct (for a detailed explanation, we kindly point the reviewer to our response to reviewer xZL3. However, we share the plots requested by the reviewer with the unnormalized performance in the following link (anonymized repo): 

https://anonymous.4open.science/r/QuickTune-F637/figures/rebuttal_figures.md

If the reviewer is satistifed with the clarifications and proposed changes we would appreciate a reflection of the discussion to the score. In case there are more questions, we are happy to answer them.

Reviewer comment (Presentation):
Instead of rank, how would Figure 1 look as a heatmap of unnormalized performance?

Author response:
We use the normalized performance since it makes the performance aggregation between the tasks correct (for a detailed explanation, we kindly point the reviewer to our response to reviewer xZL3. However, we share the plots requested by the reviewer with the unnormalized performance in the following link (anonymized repo): 

https://anonymous.4open.science/r/QuickTune-F637/figures/rebuttal_figures.md

If the reviewer is satistifed with the clarifications and proposed changes we would appreciate a reflection of the discussion to the score. In case there are more questions, we are happy to answer them.