Best Practices of Successive Halving on Neural Machine Translation and Large Language Models (2024)

Abstract: Hyperparameter optimization (HPO) enhances neural machine translation (NMT) models but demands substantial computational resources. Successive halving, a multi-fidelity HPO method, mitigates this by early stopping unpromising models and allocating more resources to promising ones. This method is particularly relevant for NMT and large language models, which are computationally intensive. However, successive halving relies on a noisy estimation of model performance and assumes that early performance is highly correlated with final performance. We introduce a table lookup benchmark dataset to study the reliability of successive halving and propose best practices for its application in NMT and large language models.

## introduction hyperparameter optimization (hpo) is crucial yet resource-intensive for transformer-based neural machine translation (nmt) models. hyperparameters such as learning rate, optimizer, batch size, and the number of nodes in each layer significantly influence the achievement of a state-of-the-art (sota) system. according to the arr responsible nlp research guidelines, presenting extensive tables of hyperparameters and the best-found values is essential in research publications. 1 recently, with the rise of large language models (llms), nmt built upon llms has shown promising results @xcite @xcite . adapting llms to nmt tasks typically involves incontext learning and supervised fine-tuning. given the abundance of parallel data, fine-tuning has proven to be more effective than in-context learning @xcite . parameter efficient finetuning (peft), such as low-rank adapter (lora, @xcite , is often favored over full finetuning due to its efficiencyfewer parameters are trained while achieving comparable or superior per-formance. despite the fixed architecture of llms during peft, new hyperparameters are introduced, including the lora rank and the specific parameters to tune, alongside traditional hyperparameters like batch size and learning rate. nmt models, whether trained from scratch or fine-tuned from llms, require extensive computational time, often taking days or weeks to converge. this makes hyperparameter searches over a reasonable space challenging. for instance, if an nmt model takes 2 gpu days to train, tuning 5 hyperparameters with 3 different values each would result in a total of 3 5 * 2 = 486 gpu days! practitioners with limited computational resources are thus often forced to resort to manual tuning or random search instead of more systematic methods like grid search or advanced hpo algorithms, increasing the risk of unfair comparisons between systems. successive halving @xcite accelerates hpo by terminating unpromising models early in a set of models trained in parallel, saving more resources with more aggressive early stopping strategies. it has shown effectiveness in computer vision @xcite and nlp tasks @xcite . however, its effectiveness for training nmt models or adapting llms for nmt tasks remains unclear. the termination decision in successive halving is heuristic, based on the ranking of model performance up to the current timestamp. it assumes that early performance is highly correlated with late performance, which may not always be true. this raises the question: does this assumption hold for nmt? if not, can we make it more reliable without relying solely on this assumption? this paper focuses on the effectiveness of successive halving for hpo in nmt models, whether trained from scratch or fine-tuned from an llm. our main contributions are summarized as follows: our findings indicate that the initial assumption of successive halving-that early performance predicts late performancegenerally holds true for nmt hpo with appropriate setups. ## successive halving the goal of successive halving @xcite is to efficiently find the optimal hyperparameter configuration within a given search space. suppose we have n configurations to explore. we begin by training all n models, and at every c checkpoints, we continue training only the top 1 p configurations based on their performance up to that point, discarding the rest. this process is repeated until only one configuration remains, which is then trained to convergence. as shown in figure 1 , we start with @xmath0. each cut is based on the best performance of the configurations up to the current checkpoint. for example, at checkpoint 10, when comparing config1 and config3, we compare config1's performance at checkpoint 5 with config3's performance at checkpoint 10. in this example, assuming it takes one gpu day (20 checkpoints) for each model to converge, successive halving can reduce the total time for hyper-parameter search from 10 days to 3.75 days. the aggressiveness of successive halving can be adjusted by changing the values of p and c. for instance, if @xmath1, the total time could be further reduced to 1.3 days. however, a more aggressive strategy increases the risk of discarding good configurations too early. in the case of @xmath2, config1 might be chosen over config2, even if con-fig2 could have performed better in the long run. ## nmtlc benchmark datasets to evaluate successive halving on nmt hpo, extensive model training until convergence is required to determine if good models are prematurely discarded. this process is resource-intensive, as each new configuration sampled for successive halving necessitates training a new nmt system from scratch. to facilitate this study, we have created a benchmark dataset that supports a table-lookup framework. we pre-train a large set of nmt systems and record their configurations and learning curves in a table. this allows for efficient evaluation of successive halving by looking up the table as needed, without training each model from scratch, significantly speeding up the experimental process. our dataset includes 2469 models trained on 9 different corpora, encompassing both models trained from scratch and those fine-tuned from llms, with a total computational cost of approximately 2519 gpu days. this is the first hpo benchmark dataset to contain nmt learning curves, enabling detailed studies on learning curves. it is also the first to include models fine-tuned from llms, facilitating hpo research on this emerging task. 1 summarizes the 6 mt corpora used for training the systems. the data cover five language pairs and three domains with varying resource levels, from low to high. ## on the reliability of successive halving on neural machine translation to evaluate the reliability of successive halving in nmt, we begin by identifying an appropriate evaluation metric (perplexity vs. bleu) for termination decisions (section 5.1). we then investigate whether halving consistently retains the best-performing half of configurations at different learning curve lengths (section 5.2.1). finally, we conduct extensive successive halving runs on random subsets of the configuration search space to assess its ability to consistently select the best configuration (section 5.2.2). ## learning curve extrapolation successive halving uses the best performance observed so far (bsf) to rank configurations at each checkpoint, assuming early performance correlates with final performance. however, as shown in figure 5 , this correlation can be low when learning curves are short. to improve on the heuristic bsf, we explore "looking forward into the predicted future" by extrapolating the optimal performance of a configuration based on partial learning curves. this predicted optimal accuracy can then be used to rank configurations more effectively in successive halving. ## conclusions successive halving is both efficient and effective for hyperparameter search in nmt tasks, significantly reducing computational resources and reliably selecting the best model with appropriate setups. however, its reliability depends on the target task and the choices of the cutting ratio (p) and cutting frequency (c). based on the studies conducted in this paper, we propose the following best practices for successive halving in nmt and llms: