An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models (2021)

Abstract: The performance of fine-tuning pre-trained language models largely depends on the hyperparameter configuration. In this paper, we investigate the performance of modern hyperparameter optimization methods (HPO) on fine-tuning pre-trained language models. First, we study and report three HPO algorithms’ performances on fine-tuning two state-of-the-art language models on the GLUE dataset. We find that using the same time budget, HPO often fails to outperform grid search due to two reasons: insufficient time budget and overfitting. We propose two general strategies and an experimental procedure to systematically troubleshoot HPO’s failure cases. By applying the procedure, we observe that HPO can succeed with more appropriate settings in the search space and time budget; however, in certain cases overfitting remains. Finally, we make suggestions for future work. Our implementation can be found in https://github.com/microsoft/FLAML/tree/main/flaml/nlp/

## introduction in the recent years, deep learning and pre-trained language models @xcite @xcite have achieved great success in the nlp community. it has now become a common practice for researchers and practitioners to fine-tune pre-trained language models in down-stream nlp tasks. for example, the huggingface transformers library @xcite was ranked no.1 among the most starred nlp libraries on github using python 1 . same as other deep learning models, the performance of fine-tuning pre-trained language models largely depends on the hyperparameter configuration. a different setting in the hyperparam-1 https://github.com/evanli/githubranking/blob/master/top100/python.md eters may cause a significant drop in the performance, turning a state-of-the-art model into a poor model. methods for tuning hyperparameters can be categorized as (1) traditional approaches such as manual tuning and grid search, and (2) automated hpo methods such as random search and bayesian optimization (bo). manual tuning often requires a large amount of manual efforts; whereas grid search often suffers from lower efficiency due to the exponential increase in time cost with the number of hyperparameters. automated hpo methods were proposed to overcome these disadvantages. recently, automated hpo methods also become increasingly popular in the nlp community @xcite . for example, bayesian optimization (bo) @xcite and population-based training @xcite both prove to be helpful for improving the performance of the transformer model @xcite for neural machine translation. the huggingface library has also added native supports for hpo in a recent update (version 3.1.0, aug 2020). with improved supports, users can now easily access a variety of hpo methods and apply them to their fine-tuning tasks. however, the effectiveness of this step is less understood. to bridge this gap, in this paper, we propose an experimental study for fine-tuning pre-trained language models using the huggingface library. this study is motivated by the following research questions: first, can automated hpo methods outperform traditional tuning method such as grid search? second, on which nlp tasks do hpo methods work better? third, if hpo does not work well, how to troubleshoot the problem and improve its performance? to answer these questions, we start from a simple initial study (section 4) by examining the performance of three hpo methods on two state-ofthe-art language models on the glue dataset. the time budget for hpo in the initial study is set to be the same as grid search. results of the initial study show that hpo often fails to match grid search's performance. the reasons for hpo's failures are two folds: first, the same budget as grid search may be too small for hpo; second, hpo overfits the task. with these observations, we propose two general strategies for troubleshooting the failure cases in hpo as well as an overall experimental procedure (figure 1 ). by applying the procedure (section 5), we find that by controlling overfitting with reduced search space and using a larger time budget, hpo has outperformed grid search in more cases. however, the overfitting problem still exists in certain tasks even when we only search for the learning rate and batch size. finally, we make suggestions for future work (section 7). the main contributions of this work are: ## definition of hpo on language model fine-tuning given a pre-trained language model, a fine-tuning task, and a dataset containing d train , d val , d test , the goal of a hyperparameter optimization algorithm is to find a hyperparameter configuration c, so that when being trained under configuration c, the model's performance on a validation set d val is optimized. formally, the goal of hpo is to find to fairly compare the performances of different hpo algorithms, the above optimization problem is defined with a constraint in the maximum running time of the hpo algorithm, which we call the time budget for the algorithm, denoted as b. under budget b, the hpo algorithm can try a number of configurations c 1 , c 2 , • • • , c n . the process of fine-tuning with configuration c i is called a trial. finally, we call the process of running an hpo algorithm a once one hpo run. ## factors of the study in this paper, we conduct an empirical study to answer the research questions in section 1. first, can automated hpo methods outperform grid search? the answer to this question depends on multiple factors, i.e., the nlp task on which hpo and grid search are evaluated, the pre-trained language model for fine tuning, the time budget, the search space for grid search and hpo algorithm, and the choice of hpo algorithm. to provide a comprehensive answer, we need to enumerate multiple settings for these factors. however, it is infeasible to enumerate all possible settings for each factor. for instance, there exist unlimited choices for the search space. to accomplish our research within reasonable computational resources 3 , for each factor, we only explore the most straight-foward settings. for example, the search space for grid search is set as the default grid configuration recommended for fine-tuning ( time budget. we focus on a low-resource scenario in this paper. to compare the performance of grid search vs. hpo, we first allocate the same time budget to hpo as grid search in our initial comparative study (section 4). if hpo does not outperform grid search, we increase the time budget for hpo. we require that each hpo run takes no more than 8 gpu hours with the nvidia tesla v100 gpu under our setting. we prune a task if the time for grid search exceeds two hours. a complete list of the time used for each remaining task can be found in pre-trained language models. in this paper, we focus on two pre-trained language models: the electra-base model @xcite , and the roberta-base model @xcite . electra and roberta are among the best-performing models on the leaderboard of glue as of jan 2021 foot_0 . another reason for choosing the two models is that they both provide a simple search space for grid search, and we find it helpful to design our hpo search space on top of them. we use both models' implementations from the transformers library @xcite ) (@xmath0.4.0). among all the different sizes of roberta and electra (large, base, small), we choose the base size, because large models do not fit into our 2-hour budget foot_1 . with the 2-hour time constraint, we prune tasks where grid search takes longer than two hours. for electra, qqp is pruned, whereas for roberta, sst, qnli, qqp, mnli are pruned. search space for grid search and hpo. it is generally difficult to design an hpo search space from scratch. in our problem, this difficulty is further amplified with the limited computational resources. fortunately, most papers on pre-trained language models recommend one or a few hyperparameter configurations for fine-tuning. we use them as the configurations for grid search. for hpo, the performance depends on the search space choice, e.g., it takes more resources to explore a large space than a smaller space close to the best configuration. due to the time budget limits, we focus on a small space surrounding the recommended grid search space, as shown in ## conclusions, discussions and future work our study suggests that for the problem of finetuning pre-trained language models, it is difficult for automated hpo methods to outperform manually tuned grid configurations with a limited time budget. however, it is possible to design a systematic procedure to troubleshoot the performance of hpo and improve the performance. we find that setting the search space appropriately per model and per task is crucial. having that setting automated for different models and tasks is beneficial to achieve the goal of automated hpo for fine-tuning. for example, one may consider automatically mining the pattern from in our study, the simple random search method stands strong against more advanced bo and early stopping methods. it suggests room for researching new hpo methods specialized for fine-tuning. a method that can robustly outperform random search with a small resource budget will be useful. it is worth mentioning that although we find hpo sometimes underperforms grid search, the grid search configurations we study are the default ones recommended by the pre-trained language models for fine tuning, therefore they may be already extensively tuned. we may not conclude that hpo is not helpful when manual tuning has not been done. how to leverage hpo methods in that scenario is an open question.