On the Interaction of Regularization Factors in Low-resource Neural Machine Translation (2022)

Abstract: We explore the roles and interactions of the hyper-parameters governing regularization, and propose a range of values applicable to low-resource neural machine translation. We demonstrate that default or recommended values for high-resource settings are not optimal for low-resource ones, and that more aggressive regularization is needed when resources are scarce, in proportion to their scarcity. We explain our observations by the generalization abilities of sharp vs. flat basins in the loss landscape of a neural network. Results for four regularization factors corroborate our claim: batch size, learning rate, dropout rate, and gradient clipping. Moreover, we show that optimal results are obtained when using several of these factors, and that our findings generalize across datasets of different sizes and languages.

## introduction the training of neural machine translation (nmt) models is governed by many hyper-parameters, which play a central role in the performances of the trained models, especially their generalization abilities. while most of the nmt frameworks recommend default values for the hyper-parameters, when it comes to low-resource settings, fewer guidelines are available. this study systematically explores the roles and interactions of a subset of hyper-parameters in low-resource nmt settings, namely those acting as regularization factors. regularizers do not fall under a single theoretical definition: @xcite view them as a collection of methods "intended to reduce generalization error but not training error." we present here a unified perspective on several regularizers which act upon the estimation of the gradients during backpropagation. using the distinction made by @xcite between flat and sharp basins in the loss landscape, we argue that noisier estimates of the gradients can increase the likelihood of finding flatter minima, which have better generalization abilities. specifically, we defend three claims: when used together, an amount of regularization from each of the four factors under study outperforms the use of any single one alone, and the best scores are robust with respect to the variation of each factor (section 8). ## data and systems we train our nmt systems with the upper sorbian (hsb) to german (de) training data of the wmt 2020 low-resource translation task @xcite . we also use the hsb-de development and test sets provided by the wmt 2020 and 2021 low-resource translation tasks @xcite , each consisting of 2k sentences. as length-based filtering does not show significant differences, we do not filter our data. additionally, in section 9, we train systems for translation from galician (gl), slovenian (sl), and slovak (sk) into english (en), using tokenized and cleaned transcriptions of ted talks @xcite . 1 finally, we train a larger german to english system using 120k lines from news commentary v13 @xcite , and sample 1,500 lines each for development and testing. we use the transformer-base architecture @xcite implemented in opennmtpy @xcite ). 3 our default setting of hyper-parameters is the one recommended by opennmt-py foot_3 which is close to the original transformer @xcite . the regularization factors appear with relatively low strengths in this setting, as is usual when large datasets are available. the setting includes the 'noam' learning rate schedule with a scaling factor of 2 and a dropout rate of 0.1. for adam, β 1 = 0.9, β 2 = 0.998 and ϵ = 10 -8 . we train our models for a maximum of 100 hours, although they generally converge earlier. when comparing batch sizes in section 4, it could be argued that epochs might provide a fairer comparison, but we measure real clock time as the most relevant measure for practitioners. a batch consists of lines (tokenized sentences) that are translated one by one, with a fixed maximum length of 512 tokens for transformer-base. lines are padded if shorter, and filtered out if longer. we train all models on two gpus with 11 gb of memory each (geforce rtx 1080ti). each device processes several batches, depending on the batch size, which are afterwards accumulated and used to update the model. the effective batch size and the batch size parameter of opennmt-py are two different values: the former is g × a × batch size, where g is the number of gpus and a the number of accumulated batches, here equal to two. 5 throughout the paper, we report the batch size parameter, but the effective batch size is in fact four times larger. we generate translations with a beam size of seven, with consecutive ensembles of four checkpoints. for each model we report the highest bleu score @xcite calculated with sacrebleu @xcite on detokenized text foot_5 as well as the chrf score @xcite . we test the statistical significance of differences in scores at the 95% confidence level using paired bootstrap resampling from sacrebleu. ## batch size in this section we train models with batch sizes ranging from 500 to 10,000, with all other hyperparameters set to default. models with batch sizes of 100 and 250 were also trained, but did not converge. the largest tested batch size is the largest value supported by our gpus. the bleu and chrf scores in there is no clear correlation between the training accuracy or cross-entropy loss and the generalization capacity, i.e. the scores on the development and test sets. the lower scores of models trained with larger batch sizes are likely not due to overfitting, because the testing curves of these models do not show any decrease late in the training. this further supports the claim that better generalization abilities are due to flat minima @xcite , section 2.1). our results are competitive with the comparable baselines from the wmt20 shared task on lowresource nmt for hsb-de @xcite , which used the same parallel data. 7 the baseline bleu scores of @xcite , @xcite and @xcite were respectively 44.1, 43.4, and 38.7 on the test set. regularization through smaller batch sizes thus provides visible improvements with respect to the default setting. larger batch sizes, however, exploit more fully the memory of the gpus, which enables higher throughput in terms of subwords processed per second, as illustrated in figure 1 , although this does not increase linearly: instead, we observe diminishing returns as batch size increases. still, while a batch size of 10k has the lowest bleu scores, it nearly doubles the throughput with respect to the highest-scoring batch size (1k). due to differences in hardware and software, these values are difficult to compare to other studies, but the trends are similar to those observed by popel and bojar (2018, section 4.1). if the regularization attained with lower batch sizes can also be obtained by using other regularization factors, this would allow the use of larger batch sizes for a more efficient training. therefore, in the next sections, we will compare a large batch size (10k) and the optimal, regularized one (1k), and verify that none of the other regularization factors that will be optimized have an impact on speed. ## learning rate previous studies by @xcite and @xcite have shown that the regularization effects of the batch size and of the learning rate may be comparable. in this section, we study the role of varying schedules of the learning rate (5.1) and the effect of resetting the schedule in midtraining, i.e. suddenly increasing the learning rate before another decrease (5.2). ## gradient clipping finally, we experiment with our fourth regularization factor: gradient clipping. since it directly involves constraining the norm of the gradient, the perspective based on flat vs. sharp basins in the loss landscape also holds for it. as in the previous sections, we compare models trained with batch sizes of 1k and 10k, but observe no statistically significant differences between them when using default values for other hyper-parameters, with bleu scores shown in ## combining regularization factors we will now show that a combination of regularization factors can produce higher scores than individual factors used separately, and that the maximal scores are stable when varying the strengths of regularizers around their optimal values. the batch size is fixed at 10k, since this enables a higher training speed than 1k with similar best scores, provided that other regularization factors are used, as shown in tables 3, 4 and 5. the number of warmup steps is fixed at 16k since we showed in section 5.1 that this parameter mainly limits the peaks of the learning rate and thus prevents models from diverging early in the training. our search space for the other regularization factors is shown in the comparison of scores in ## testing on additional corpora in this section, we confirm our claims using additional low-resource datasets. we consider two smaller samples with 40k and 20k lines from the hsb-de corpus, as well as parallel datasets for galician, german, slovak and slovenian (see section 3). we do not optimize regularization factors on each dataset, but only use the optimal hyper-parameters found above on hsb-de with 60k lines. ## conclusion we presented a unified perspective on the role of four regularization factors in low-resource settings: batch size, learning schedule, gradient clipping and dropout rate. the results support our claim that more regularization is beneficial in such settings, with respect to the default values that are recommended for high-resource settings. we first substantiated the claim for each factor taken individually, and then showed that a combination of factors leads to improved scores and is robust when factors vary. finally, we showed that our findings generalize across different low-resource sizes and languages. overall, we interpreted the results from the perspective of the loss landscape, and argued that more regularization is beneficial because the noise it introduces in the estimation of gradients leads to finding flatter minima of the loss, which have better generalization abilities. we hope that better insights on the loss landscape of the transformer will confirm our theoretical interpretation, and that the observations put forward in this paper will also help practitioners with setting hyperparameters for low-resource nmt systems.