Paper: A SYSTEMATIC STUDY ON EARLY STOPPING CRITERIA IN HPO AND THE IMPLICATIONS OF UNCERTAINTY (ICLR-2024)

Reviewer comment (Experiments):
Only the Hyperband algorithm, which was published six years ago, was examined in evaluating the utility of early stopping criteria in hyperparameter optimization. More state-of-the-art early stopping criteria should be investigated and analyzed to draw a convincing conclusion.

Author response:
**New HPO algorithm**

Thanks for the suggestion. We further conduct experiments with two widely recognized algorithms: BOHB [1] and the state-of-the-art Sub-sampling [2]. The results and detailed settings can also be found in Appendices B and E in the updated version of our paper. It can be seen that the training loss significantly outperforms other metrics across all HPO algorithms. Furthermore, introducing uncertainty into metrics demonstrates enhanced reliability—an aspect we delve into within our findings.

[1] https://arxiv.org/abs/1807.01774

[2] https://arxiv.org/abs/2209.12499

Reviewer comment (Presentation):
What is the fraction of budget shown in the figures? What is its relation with the filtering ratios?

Author response:
**Terminology**

Thanks for raising this concern. We clarify the terminology used in our work as follows:

1. "Gaps" in Insight 2 refers to the differences observed in performance metrics across various models, essentially reflecting the concept of performance regret. A more discriminative metric leads to clearer differences between model performances, thereby enhancing reliability.

2. The lower plots in Figure 1 depict the discrepancy between the final test accuracy achievable by the remaining model candidates after each instance of early stopping and the best final test accuracy achievable by the whole set of samples as training progresses. The term "fraction of budget" denotes the proportion of allocated budget utilized during training. These results are derived from a scenario where R=150 and the filtering ratio equals 3. Each data point within these plots signifies a round of early stopping. The "fraction of budget" serves as an indicator of the early stopping point, specifically denoting the epoch at which early stopping is executed, calculated based on the filtering ratio ($\eta$) and R. Furthermore, the count of retained model candidates is determined according to the filtering ratio in each early stopping iteration.

We have added explanations in Section 2 and Insight 2.

Reviewer comment (Writing):
What are the answers to the three fundamental questions given in the Introduction?

Author response:
**Answers to the fundamental questions**

This paper addresses the three fundamental questions across Sections 2 to 4, providing distinct insights into each:

1. Section 2 empirically compares commonly used criteria. It reveals that different criteria may have significantly different impacts on the HPO performance, and training loss tends to suit complex tasks better than validation loss. 

2. Section 3 analyzes the factors that influence early stopping, offering insights for Section 2's findings. The nature of early stopping revolves around the idea of monitoring a model's performance during training and halting the training process when certain criteria or performance metrics are met. This practice essentially involves “a trade-off between the HPO cost and the effectiveness of the final models”. Criteria or metrics are responsible for estimating and ranking the models' capabilities. According to Section 3, the factors that affect the validity of a criterion are the size of the dataset on which the calculation is based and whether or not the criterion contains uncertainty that affects discrimination. As we analyzed in Section 3, “while a larger training dataset may contribute, the primary justification lies in the training loss serving as a superior indicator of model expressiveness, yielding greater stability and consistent model ranking. In contrast, the validation loss prioritizes the aspect of generalization and tends to favor models with lower capabilities that converge early. ”

3. Section 4 investigates the impact of model uncertainty on early stopping. Besides the commonly considered measures, other early stopping criteria include the combination of training and validation metrics validated in Section 3 and Appendix D, and a collection of methods that incorporate uncertainty explored in Section 4. Model uncertainty has some important impact on the effectiveness of early stopping. As stated in Section 4.1, “if an early stopping decision is made at a point of peak uncertainty, stronger models may be prematurely terminated.” By examining a series of criteria that introduce uncertainty, this paper concludes that “a careful combination of model uncertainty with conventional early stopping criteria can yield significant benefits for HPO, especially when dynamic trends of the training process are considered.”

Thanks for raising this question. We have given a summary of our findings near the Introduction for readers to grasp the message.

Reviewer comment (Novelty):
The overall contribution and relevance of the paper are unclear, and there is confusion about some of its technical aspects.

Author response:
Thanks for raising this valuable question. This work belongs to an insight paper, aiming at exploring the impact of diverse performance metrics within the early stopping mechanisms of HPO algorithms. Our experiments centered around commonly used metrics, including training/validation loss/accuracy. The results highlighted the superior efficacy of training loss, especially in intricate tasks with slower convergence. This finding is intriguing, considering that validation loss has been widely adopted. We substantiated this observation through experiments and analysis, identifying factors - model uncertainty and dataset size - being potential reasons for the inferior performance of validation metrics. Furthermore, we explored strategies leveraging uncertainty to enhance the reliability of these metrics.

We have followed your advice to improve the organization of this paper. In particular, we have given a summary of our findings in the Introduction for readers to grasp the message. We also revised some sentences in Section 1 to make our motivation clearer. We've expanded the experimental scope in Appendices A, B, and E, introducing new benchmarks (MLP and LogReg from HPOBench) and incorporating additional HPO algorithms (BOHB and Sub-sampling), to enrich the data basis of this work.

Reviewer comment (Novelty):
The paper lacks a clear contribution and coherent narrative ("red path"). It reads like a collection of experiments exploring aspects of uncertainty in HPO and methods analysis, but most insights are intuitive and expected, lacking rigor. The nature of the contribution lies ambiguously between an insight paper and a technical proposal, and in both cases, it falls short—insight papers require more extensive data and less vague insights, while technical papers require stronger comparisons.

Author response:
Thanks for raising this valuable question. This work belongs to an insight paper, aiming at exploring the impact of diverse performance metrics within the early stopping mechanisms of HPO algorithms. Our experiments centered around commonly used metrics, including training/validation loss/accuracy. The results highlighted the superior efficacy of training loss, especially in intricate tasks with slower convergence. This finding is intriguing, considering that validation loss has been widely adopted. We substantiated this observation through experiments and analysis, identifying factors - model uncertainty and dataset size - being potential reasons for the inferior performance of validation metrics. Furthermore, we explored strategies leveraging uncertainty to enhance the reliability of these metrics.

We have followed your advice to improve the organization of this paper. In particular, we have given a summary of our findings in the Introduction for readers to grasp the message. We also revised some sentences in Section 1 to make our motivation clearer. We've expanded the experimental scope in Appendices A, B, and E, introducing new benchmarks (MLP and LogReg from HPOBench) and incorporating additional HPO algorithms (BOHB and Sub-sampling), to enrich the data basis of this work.

Reviewer comment (Writing):
In Section 2, it is not clear what the authors identify as the decision variable for early discarding. The use of "criterion" is potentially misleading, as "performance metric" or "performance measure" might be more appropriate. The motivation for the four cases studied is unclear, and the explanation of "reliability of criteria" is ambiguous. The reasoning behind using test accuracy as the eventual objective is not well justified, especially if the loss is cross-entropy (log-loss). Additionally, the selection of datasets for Fig. 1 is unclear, particularly why some LCBench datasets are omitted.

Author response:
We fully agree with you that in most cases the metrics are determined by user needs. As we pointed out in Section 1, the choice of metrics within HPO is based on the practitioners' personal preferences. Regarding decision variables, our study has shown that delving into the reliability of metrics yields benefits not only for users but also for HPO tools. From the user's perspective, if there is a priori knowledge of a model's convergence behavior, one can choose between training and validation losses based on whether overfitting occurs at early stopping points. Specifically, if there is overfitting, validation loss is preferred; otherwise, it is the training loss. For the HPO tools, they can request both training and validation losses from users, thus incorporating an overfitting detection to avoid unfair comparisons, and making decisions based on which metric contains less uncertainty. Additionally, they can introduce uncertainty, as shown in Section 4, to design more robust metrics for more reliable decision-making. We have outlined general guidelines in Section 5.

Regarding the motivation of this paper, we have followed your suggestion to replace "criterion" with "performance metric" and made a clear distinction in Section 1. In addition to comparing loss and external metrics (e.g., accuracy), we also seek to compare the differences between training and validation metrics. We have modified the expressions in Section 1 and Insight 1 to make the motivation clearer.

Reviewer comment (Theory):
In the theoretical part, it is unclear whether D refers to training or validation data, and the meaning changes depending on whether training/validation accuracy/loss is computed. There is a logical flaw in arguing that tighter bounds on regret imply lower regret, as this is only conjectural. The theoretical statements also appear contradictory to Section 2’s observations; for example, large datasets such as CIFAR and ImageNet should, if anything, provide more reliable performance criteria, yet the findings do not align with this expectation.

Author response:
We fully agree with the comment that D can be a training or validation set, and that model complexity plays a role in HPO efficacy. However, we would like to clarify that our discussion is in the context of "one early-stopping based HPO", which implies a fixed DL task, similar model complexity (with different hyperparameters), and fixed training, validation, and testing sets. Our goal is to allow users or HPO tools to select the best metric when faced with an HPO task, where there is no need to compare between multiple HPO tasks. We have added a detailed description in Section 3 to provide further clarity.

Section 3 aims to delve into strategies for minimizing decision errors, which we define as the probability of the optimal model's loss being greater than the loss of a suboptimal model. By minimizing the probability, it supports the conclusion that "using a larger dataset or choosing more discriminative metrics can yield more reliable results". The training metrics stand out due to their foundation on a larger database, offering more stable values, as demonstrated in our experiments on Nas-Bench-201 and the new benchmarks outlined in Appendix A, B, and E. While validation metrics outperform training metrics on certain datasets like Fashion-MNIST and jasmine, this discrepancy does not contradict our statements. Instead, as explained in the second paragraph of Section 3, it stems from lower model complexity leading to overfitting.

We have refined Insight 1 and highlighted certain statements in Section 3 to enhance the clarity of our findings. We have also updated Section 4.2.1 and Appendices A and E with additional technical details.