Reproducible and Efficient Benchmarks for Hyperparameter Optimization of Neural Machine Translation Systems (2020)

Abstract: Hyperparameter selection is a crucial part of building neural machine translation (NMT) systems across both academia and industry. Fine-grained adjustments to a model’s architecture or training recipe can mean the difference between a positive and negative research result or between a state-of-the-art and underperforming system. While recent literature has proposed methods for automatic hyperparameter optimization (HPO), there has been limited work on applying these methods to neural machine translation (NMT), due in part to the high costs associated with experiments that train large numbers of model variants. To facilitate research in this space, we introduce a lookup-based approach that uses a library of pre-trained models for fast, low cost HPO experimentation. Our contributions include (1) the release of a large collection of trained NMT models covering a wide range of hyperparameters, (2) the proposal of targeted metrics for evaluating HPO methods on NMT, and (3) a reproducible benchmark of several HPO methods against our model library, including novel graph-based and multiobjective methods.

## introduction choosing effective hyperparameters is crucial for building strong neural machine translation (nmt) systems. although some choices present obvious trade-offs (e.g., more and larger layers tend to increase quality at the cost of speed), others are more subtle (e.g., effects of batch size, learning rate, and normalization techniques on different layer types). optimal versus suboptimal hyperparameters can lead to dramatic swings in system performance; consider the wide range of bleu scores for variants of the same base system in figure 1 (left). in practice, these hyperparame-ters are often tuned manually based on intuition and heuristics, a tedious and error-prone process that can lead to unreliable experimental results and underperforming shared task or production systems. the difficulty is compounded when system builders must jointly optimize multiple objectives, such as translation accuracy (bleu) and decoding speed, which are largely uncorrelated as shown in figure 1 (right). in the past decade, various hyperparameter optimization (hpo) methods have emerged in the machine learning literature under the labels of ''automl'' @xcite @xcite and ''neural architecture search'' @xcite @xcite . however, it is unclear how they perform on nmt; we are not aware of any prior work with comprehensive evaluation. one challenge is that the state-of-the-art nmt models @xcite @xcite require significant computational resources for training. secondly, they usually have large hyperparameter search spaces. thus, it is prohibitively expensive in practice to compare hpo methods on nmt tasks. in order to enable reproducible hpo research on nmt tasks, we adopt a benchmark procedure based on ''table-lookup''. this approach was recently introduced to neural architecture search by @xcite , and to hyperparameter optimization by @xcite . first, we train an extremely large number of nmt models with diverse hyperparameter settings and record their performance metrics (e.g., bleu, decoding time) in a table. then, we constrain our hpo methods to sample from this finite set of models. we have trained and pre-computed a large number of representative nmt models beforehand, hpo algorithm developers no longer need to deal with the cost of training nmt. importantly, this kind of benchmark significantly speeds up the hpo experiment turnover time, enabling fast repeated trials for rigorous tests and facilitates detailed error analysis. the main contributions of this work are: ## hpo problem definition given a machine learning algorithm with h hyperparameters, we denote the domain of the one class of algorithms that tackles the hpo problem is sequential model-based optimization (smbo), illustrated in figure 2 . smbo approximates f with a cheap-to-evaluate surrogate model f @xcite @xcite . smbo starts by querying f with initial hyperparameters {λ init } and recording the resulting (λ init , f (λ init )) pairs. then, it iteratively (1) fits the surrogate f on pairs observed so far; (2) gets the predictions f (λ i ) for unlabeled/unobserved hyperparameters; and ## evaluation protocols to assess hpo method performance, we measure the runtime to reach a quality indicator (e.g., bleu) target value. the runtime is defined as the number of nmt models trained, or equivalently the number of function evaluations f (λ) in figure 2 . we consider two ways to measure the hpo performance: fixed-target and fixedbudget. ## methods we now describe two hpo/smbo methods used in our experiments: bayesian optimization 10 is a popular method. graph-based smbo is a novel method that adapts ideas in graph-based semi-supervised learning to the hpo problem. ## experiments and results we evaluate hpo methods on six nmt tasks with the provided benchmark dataset and report their performance measured by three runtime-based assessment metrics mentioned in section 4. the code base is provided to ensure reproducibility. 14 ## related work to alleviate the computational burden for benchmarking hpo methods and to improve research reproducibility, several studies have explored the table-lookup framework. @xcite published a mix of datasets focusing on feed forward neural networks. @xcite released a dataset of convolutional architectures for image classification problems. to the best of our knowledge, this work is the first that focuses on nmt and transformer models. one challenge with table-lookup is that sufficient coverage of the hyperparameter grid is assumed. @xcite and @xcite propose using a predictive metamodel trained on a table-lookup benchmark to approximate hyperparameters that are not in the table. this is an interesting avenue for future work. studies on hpo for nmt are scarce. @xcite propose an evolution strategy-based hpo method for nmt. @xcite apply nas to transformer on nmt tasks. there is also work on empirically exploring hyperparameters and architectures of nmt systems @xcite @xcite , though the focus is on finding general best-practice configurations. this differs from the goal of hpo, which aims to find the best configuration specific to a given dataset. ## conclusions in this paper, we presented a benchmark dataset for hyperparameter optimization of neural machine translation systems. we provided multiple evaluation protocols and analysis approaches for comparing hpo methods. we benchmarked bayesian optimization and a novel graph-based semi-supervised learning method on the dataset for both single-objective and multiobjective optimization. our hope is that this kind of dataset will facilitate reproducible research and rigorous evaluation of hpo for complex and expensive models.