Should I try multiple optimizers when fine-tuning a pre-trained Transformer for NLP tasks? Should I tune their hyperparameters? (2024)

Abstract: NLP research has explored different neural model architectures and sizes, datasets, training objectives, and transfer learning techniques. However, the choice of optimizer during training has not been explored as extensively. Typically, some variant of Stochastic Gradient Descent (SGD) is employed, selected among numerous variants, using unclear criteria, often with minimal or no tuning of the optimizer’s hyperparameters. Experimenting with five GLUE datasets, two models (DistilBERT and DistilRoBERTa), and seven popular optimizers (SGD, SGD with Momentum, Adam, AdaMax, Nadam, AdamW, and AdaBound), we find that when the hyperparameters of the optimizers are tuned, there is no substantial difference in test performance across the five more elaborate (adaptive) optimizers, despite differences in training loss. Furthermore, tuning just the learning rate is in most cases as good as tuning all the hyperparameters. Hence, we recommend picking any of the best-behaved adaptive optimizers (e.g., Adam) and tuning only its learning rate. When no hyperparameter can be tuned, SGD with Momentum is the best choice.

## introduction nlp research has investigated how different neural model architectures (e.g., rnns, cnns, transformers), model sizes, datasets, training (or pretraining) objectives, and transfer-learning techniques (e.g., pre-training and fine-tuning) affect performance and efficiency. however, the effects of using different optimizers to minimize the training loss have not been explored as extensively. adam (kingma and ba, 2014) is a popular choice, but there are dozens of alternatives, mostly variants of stochastic gradient descent (sgd) @xcite ). 1 optimizer selection seems to be based on unclear criteria and anecdotal information. furthermore, most optimizers have several hyperparameters, often minimally tuned (e.g., only the learning rate is tuned) or left to their default values. hence, when models need to be trained (e.g., pre-trained or finetuned), it is unclear if the available computing resources should be used to try multiple optimizers, tune their hyperparameters, both, or none. our work is inspired by @xcite , who experimented with 15 optimizers and 8 tasks from deepobs @xcite . their most striking finding was that trying several optimizers with default hyperparameters was almost as beneficial as (and cheaper than) picking any single (competent) optimizer and tuning its hyperparameters. hence, practitioners would be advised to try multiple optimizers with defaults, rather than selecting a single optimizer (e.g., based on anecdotal evidence) and tuning its hyperparameters, when they cannot tune both the choice of optimizer and hyperparameters (which is expensive). in fact, tuning the hyperparameters of a single optimizer was only slightly better than using its defaults, which also advocates against hyperparameter tuning. however, @xcite considered only one nlp task (character-level language modeling) with an rnn, acknowledging that their findings may not hold with more complicated models, such as transformers. they also found indications that the best optimizer may depend on the model and task. we complement the work of @xcite from an nlp perspective by investigating empirically if it is worth (a) trying multiple optimiz-ers and/or (b) tuning their hyperparameters (and which ones), when fine-tuning pre-trained transformer encoders. we experiment with five tasks from glue @xcite , using seven popular optimizers, namely sgd, sgd with momentum (sgdm) @xcite , adam and adamax @xcite , nadam @xcite , adamw @xcite , and ad-abound @xcite . for each task and optimizer, we fine-tune distilbert @xcite and distilroberta, 2 two efficient pre-trained transformers that allow us to complete the many experiments of this work with our limited budget. we consider three cases: using the default hyperparameters of the optimizers, tuning all their hyperparameters, or tuning only their learning rates. with the exception of the two non-adaptive optimizers considered, i.e., plain sgd and sgdm, which are largely unaffected by hyperparameter tuning, the test performance of the other five (adaptive) optimizers improves substantially when their hyperparameters are tuned, unlike smaller overall improvements reported by @xcite . interestingly, in most cases tuning only the learning rate is as good as (and much cheaper than) tuning all the hyperparameters. furthermore, when hyperparameters (or just the learning rate) are tuned, the adaptive optimizers have very similar test scores in most cases, unlike plain sgd and sgdm, which are clearly the worst and second worst, respectively. this parity of test performance of the adaptive optimizers is obtained despite occasional differences in the training loss they reach. when no hyperparameter can be tuned (e.g., due to limited budget), sgdm is the best choice and adabound occasionally works relatively well, but the other optimizers considered are much worse. trying multiple optimizers with defaults @xcite is reasonably good too, because of the good untuned performance of sgdm and adabound. however, our experiments suggest that picking just one of the best-behaved adaptive optimizers i.e., with consistent top-performance across datasets, e.g., adam, and tuning only its learning rate is the best strategy. ## optimizers considered all the optimizers we consider aim at tuning the weights vector θ of a model, so that a loss function the step could be towards the opposite direction of g(θ). however, exactly computing g(θ) at each step is too costly for large training set sizes k. ## related work deepobs @xcite is an optimizer benchmarking suite that includes several classical datasets, models (e.g., cnns, rnns), optimizer implementations (currently sgd, sgdm, adam), and facilities to compare optimizers. however, it includes only one nlp task (character-level language modeling with an rnn) and no transformer models. a similar observation can be made for the more recent algoperf benchmark @xcite , which includes transformers, but only one nlp task (machine translation). multitask @xcite ) also considers only three nlp tasks (language modelling with characters or words/subwords, text classification), all with rnns. as already noted, we were inspired by @xcite , who experimented with 15 optimizers and 8 tasks (from deepobs), but only one nlp task (the only one of deepobs), without considering transformers. they found that adam remained a strong contender, with more recent variants failing to consistently outperform it. tuning the hyperparameters of a single optimizer was overall only slightly better than using its default hyperparameter values (median improvement 3.4% for a tuning budget of 50 trials and diminishing returns for larger budgets). trying several optimizers with defaults was almost as beneficial as (and cheaper than) picking any single (competent) optimizer and tuning it. schmidt et al. acknowledged, however, that their findings may not hold with more complicated models, such as transformers. they also found indications that the best optimizer may depend on the model and task. they employed random search for hyperparameter tuning, whereas we used optuna (section 3.2). they also experimented with four update schedules for the learning rate (constant, cosine decay, cosine with warm restarts, and trapezoidal) on top of the tuned rate, with results indicating that non-constant schedules add small gains; we considered only a constant schedule. @xcite reported that adaptive optimizers may lead to worse development or test performance than sgd, even in cases where the adaptive optimizers reach lower training losses. they considered artificial datasets, an image classification task (using a cnn), character-level language modelling, and two parsing tasks, the latter three tasks using lstm-based models. however, they only tuned the learning rate and the learning rate decay scheme, as pointed out by @xcite . @xcite introduced the notion of inclusion between optimizers. for example, sgd is a special case of (is included by) sgdm for α = 0 (algorithm 1). with an exhaustive hyperparameter search, an optimizer should never perform worse than an optimizer it includes. indeed, choi et al. show that with extensive hyperparameter tuning, inclusion relationships reflect end-task performance, and they criticize previous work by @xcite and @xcite for not having tuned all hyperparameters. we tuned all hyperparameters, but found that tuning only the learning rate was equally good. we also note that extensive tuning of the kind recommended by @xcite coarser to finer swaps to explore and define the search space anew per task) is computationally very expensive. finally, we note that @xcite considered only image classification and language modelling, the latter with an lstm and a (not pre-trained) transformer. sivaprasad et al. ( 2020 ) pointed out that when comparing optimizers, it is important to consider how easy it is to reach reasonable performance with a limited budget (number of trials), rather than comparing performance scores obtained with very extensive (and costly) hyperparameter tuning, unlike the setting of @xcite . they experimented with sgdm, adagrad @xcite , adam, and adamw, with a range of budgets, in nine tasks, of which only two were nlp tasks (sentiment analysis, news classification), without considering transformers. they recommended using adam and tuning only its learning rate, especially with low budgets, which agrees with our conclusions. although we experimented with a single, relatively small budget (30 trials), we used the same budget for all optimizers, like sivaprasad et al. (2020) their evaluation protocol, which efficiently simulates multiple budgets, could be used in future extensions of our work, though it is incompatible with our use of optuna, as it requires random search. a particularly interesting research direction is to semi-automatically discover new optimization algorithms via evolutionary program searchs and manual intervention. @xcite recently used this approach to produce lion, an optimizer that, among other experiments, was reported to be overall slightly better than adamw on glue, when using the t5 model @xcite , but tuning only the learning rates and decoupled weight decay hyperparameters of the two optimizers. the optimizers we considered are first-order, i.e., they compute only the gradient of the loss function and not its hessian (second-order partial derivatives), which would be prohibitively costly (m 2 second-order derivatives at each step, for a model with m weights). interestingly, @xcite investigate an approximate second-order optimizer for use in language model pretraining. furthermore, the optimizers we considered use a single value of the learning rate at each step. backtracking methods consider multiple learning rate values at each step, computing the loss for each one. although backtracking is very common in traditional optimization, apparently it has not received sufficient attention in machine learning; an exception is the work of truong and nguyen (2021). ## conclusions we investigated if it is worth (a) trying multiple optimizers and/or (b) tuning their hyperparameters (and which ones), when fine-tuning a pre-trained transformer. we experimented with five glue datasets, two efficient pre-trained transformer encoders (distilbert, distilroberta), and seven popular optimizers (sgd, sgdm, adam, adamax, nadam, adamw, and adabound). with the exception of the two non-adaptive optimizers (sgd, sgdm), which were largely unaffected by hyperparameter tuning, the test performance of the other five (adaptive) optimizers improved substantially when they were tuned, unlike previously reported smaller overall gains @xcite . in most cases, tuning only the learning rate was as good as (and cheaper than) tuning all the hyperparameters. furthermore, when hyperparameters (or just the learning rate) were tuned, all the adaptive optimizers had very similar test scores, unlike sgd and sgdm, which were clearly the worst and second worst, respectively. this parity of test performance of the adaptive optimizers was obtained despite occasional differences in the training loss they reached. when no hyperparameter was tuned (which might be the case with a low budget), sgdm was the best choice and adabound the second best; the other optimizers were much worse. trying multiple optimizers with defaults @xcite worked reasonably well too, but only because of the good untuned performance of sgdm and (to a lesser extent) adabound; trying the other optimizers untuned would have been a waste of resources. hence, we suggest picking just one adaptive optimizer and tuning only its learning rate; we recommend adamw, nadam, or adam, which were consistently top performers when tuning only the learning rate. our work can help save substantial effort, computational resources, and energy, by reducing the number of hyperparameter tuning experiments practitioners perform. however, our findings need to be complemented by future additional experiments with more models, more nlp tasks (including pre-training tasks), and different tuning budgets (i.e., different maximum number of trials); see also section 6. this is a computationally very expensive exploration that might be best handled by multiple groups performing and reporting similar studies. towards this direction, we make all the code, data, and results of our experiments publicly available. 8