Bayesian Optimization (BO) is a successful methodology to tune the hyperparametersof machine learning algorithms. The user defines a metric of interest, such as the validation error, and BO finds the optimal hyperparametersthat minimize it. However, the metric improvements on the validation set may not translate to the test set, especially  on small datasets. In other words,  BO can overfit. While cross-validation mitigates this, it comes with high computational cost. In this paper, we carry out the first systematic investigation of overfitting in BO and demonstrate that this is a serious yet often overlooked concern in practice. We propose the first problem-adaptive and interpretable criterion to early stop BO, reducing overfitting while mitigating the cost of cross-validation. Experimental results on real-world hyperparameter optimization tasks show that our approach can substantially reduce compute time with little to no loss of test accuracy,demonstrating a clear practical advantage over existing techniques .
Tuning machine learning models with Bayesian optimization (BO) is a successful strategy to find good hyperparameters. BO defines an iterative procedure where a cross-validated metric is evaluated on promising hyperparameters. In practice, however, an improvement of the validation metric may not translate in better predictive performance on a test set, especially when tuning models trained on small datasets. In other words, unlike conventional wisdom dictates, BO can overfit. In this paper, we carry out the first systematic investigation of overfitting in BO and demonstrate that this issue is serious, yet often overlooked in practice. We propose a novel criterion to early stop BO, which aims to maintain the solution quality while saving the unnecessary iterations that can lead to overfitting. Experiments on real-world hyperparameter optimization problems show that our approach effectively meets these goals and is more adaptive comparing to baselines .
R
Bayesian Optimization
Tuning machine learning models with Bayesian optimization
R
methodology to tune the hyperparametersof machine learning algorithms. The user defines a
strategy to find good hyperparameters. BO defines an iterative procedure where a cross-validated metric is evaluated on promising hyperparameters. In practice, however, an improvement of the validation
D
of interest, such as the validation error, and BO finds the optimal hyperparametersthat minimize it. However, the metric improvements on the validation set
R
to the
in better predictive performance on a
A
when tuning models trained
A
unlike conventional wisdom dictates,
D
While cross-validation mitigates this, it comes with high computational cost.
R
is a serious
issue is serious,
D
concern
R
the first problem-adaptive and interpretable
a novel
R
reducing overfitting while mitigating the cost of cross-validation. Experimental results
which aims to maintain the solution quality while saving the unnecessary iterations that can lead to overfitting. Experiments
R
tasks
problems
R
can substantially reduce compute time with little to no loss of test accuracy,demonstrating a clear practical advantage over existing techniques
effectively meets these goals and is more adaptive comparing to baselines