Small Batch Sizes Improve Training of Low-Resource Neural MT (2021)

Abstract: We study the role of an essential hyper-parameter that governs the training of Transformers for neural machine translation in a low-resource setting: the batch size. Using theoretical insights and experimental evidence, we argue against the widespread belief that batch size should be set as large as allowed by the memory of the GPUs. We show that in a low-resource setting, a smaller batch size leads to higher scores in a shorter training time, and argue that this is due to better regularization of the gradients during training.

## introduction training transformers for low-resource neural machine translation (nmt), i.e. when only small parallel corpora are available, raises the challenge of finding optimal hyper-parameters. while several fixed configurations of the transformer @xcite have been empirically validated by the community, such as 'base' or 'big', the settings of many other hyper-parameters rely on tips from practitioners. however, these values are not always suitable to low-resource settings, and systematic studies in these settings are rare @xcite . in this paper, we show that the best values of a hyper-parameter that is essential for training, namely batch size, differ in low-resource settings from those commonly accepted when larger data sets are available. we analyze the role of small batch sizes, inspired by studies in computer vision @xcite , and then pinpoint empirically the optimal trade-off between a high batch size (for efficiency) and a small one (for regularization). although large batch sizes were found to lead to higher-quality models in experiments with highresource nmt @xcite , we show here that smaller batch sizes can outperform the latter, likely due to a regularizing effect in the gradient update. moreover, we show that this finding is invariant to changes in tokenization methods. the paper is organized as follows. in section 2, we discuss batch size from a machine learning perspective, showing why smaller values of batch size may act as regularizers. then, in section 3, we review studies of hyper-parameters in nmt. in section 4, we present the parameters of our transformer and the data from the wmt 2020 lowresource task @xcite and other sources that we use in our experiments. in section 5, we provide empirical evidence that smaller batch sizes are preferable in low-resource settings. ## ml perspective on batch size machine learning theory argues that performing back-propagation with large batch sizes leads to better optimization, because the estimates of the gradients are more accurate. conversely, using small batches during training leads to noisier gradient estimations, i.e. with a larger variance in comparison to the gradient computed over the entire training set. still, one advantage of small batch sizes is that they are more likely to make parameters converge towards flatter minima of the loss @xcite , as explained below. such flatter minima have better generalization capacities, i.e. they maintain performance when presented with a new test set. @xcite define a flat minimizer -as opposed to a sharp one -as a point in the parameter space that is a local minimum of the loss function, and where this function varies slowly in a relatively large neighborhood. @xcite point to the following generalization gap: training with large batch sizes tends to converge towards sharp minimizers, which offer poorer generalization ca-pacities. conversely, small batch sizes allow convergence towards flat minimizers, which are likely to generalize better. thus, smaller batch sizes have exploration abilities: the search is more likely to exit the basins of sharp minimizers, and to tend towards flat minimizers, from where noise will not cause them to exit. since a sharp minimizer requires high precision to be described, unlike a flat one, the more noise there is in the gradient, the more unlikely it is that the parameters will converge towards a sharp minimizer. this is precisely the contribution of a smaller batch size: introduce noise in the gradient estimation. according to this theoretical view, above a certain threshold of the batch size, the generalization capacities of a model deteriorate. the threshold depends on several hyper-parameters, including the batch size. its role has not been fully settled yet, with observations and conclusions varying widely across studies @xcite @xcite @xcite . moreover, these studies are on image data sets, with fully connected or with convolutional nns, which differ substantially from nmt settings. ## the role of batch size in neural mt several recent studies in nmt have considered batch size among other hyper-parameters, but they have either been in high-resource settings @xcite or have given only marginal attention to batch size @xcite . @xcite reported that bleu scores increased with batch size (including when using more gpus) in a transformer-based nmt system, although with diminishing returns, recommending in particular that "batch size should be set as high as possible". their experiments were performed using mainly two datasets, with respectively 58m and 15m sentence pairs. it thus remains an open question whether their findings regarding batch size also apply when much less training data is available. @xcite experimented with a recurrent network in a low-resource setting and found that smaller batch sizes were beneficial, along with other forms of regularization. they experimented with two batch sizes of 4,000 and 1,000 tokens, and observed improvements with the latter of 0.30 and 0.04 bleu points on data sets with 5k and 160k sentence pairs, respectively. it is difficult to predict from these results what the optimal batch size is for transformer-based nmt. araabi and monz (2020) studied the role of 15 hyper-parameters of the transformer, with several sizes of low-resource datasets. for the largest training sizes tested (80k and 165k sentence pairs), larger batch sizes improved performance, with respectively 8,192 and 12,288 versus 4,096 for the other sizes. for lower training sizes, smaller batch sizes did not improve performance, which the authors explain by transformer's need for larger batches. in our view, an alternative explanation is the order of optimization of the hyper-parameters (a grid search in which they optimize one hyperparameter at a time): batch size is #12 out of 15, so by the time several sizes are compared, regularization has already been introduced in the model by dropouts on words, activation, and layers. late optimization of batch size, of warmup steps (#14) or of learning rate (#15) cannot properly determine their regularizing effects. @xcite proposed to compute gradients while accumulating minibatches, and observed that increasing batch size stabilizes gradient direction up to a certain point, after which it starts to fluctuate. they used this criterion to dynamically adjust batch sizes while training. in their experiments with large training sets (4.5m and 36m sentence pairs), their average batch size was around 26k on two gpus, and never lower than 7k. their observations on the gradient direction as more minibatches are accumulated are consistent with the findings of @xcite who see diminishing returns when increasing batch size. ## datasets and systems we train nmt systems with two low-resource parallel corpora, listed in the first two lines of we build subword vocabularies using the unigram lm model @xcite as implemented in sentencepiece foot_0 , with the monolingual corpora from to train the sentencepiece model for the de-en, for comparison purposes, we treat german as a low-resource language, and sample 680k lines of english and german from europarl v3 @xcite , which we combine respectively with the 60k lines extracted from the de-en parallel corpus. we use the transformer-base @xcite in the implementation provided by open-nmt @xcite , with the parameters given in appendix a. unless otherwise specified, we follow opennmt-py's recommended values for the hyper-parameters. 2 when using several gpus with gradient accumulation, each gpu processes several batches, which are then accumulated across all gpus and used to update the model at each step. therefore, the effective batch size is b × g × a, where b is the individual batch size, g is the number of gpus and a the number of accumulated batches, and differs from the batch size hyper-parameter b. we train all models on two geforce rtx 1080ti gpus with 11 gb of memory each and accumulate gradients over two minibatches (@xmath1, following opennmt-py's recommendation. therefore, the batch size parameter is not our effective batch size, which is four times larger. throughout this work, we will refer to batch size b as the batch size parameter, and report true epochs, which we define as computed with the effective batch size as s × b eff /n , for s training steps, b eff effective batch size, and n number of source tokens in the training set. following opennmt-py's recommendations, we set the adam hyper-parameters at β 1 = 0.9, β 2 = 0.998, = 10 -8 and apply at each step a scaling factor of two to noam's learning rate schedule, setting warmup steps to 8k. translations are generated with a beam width of seven, with an ensemble of the last four saved checkpoints. we report bleu scores @xcite obtained with sacrebleu @xcite on detokenized text. ## experimental results to study the impact of batch sizes in a low-resource setting, we train various hsb-de and de-en models for 700 epochs with the following batch sizes: 100, 250, 500, 1,000, 2,500, 5,000, 7,500, 10,000, and 10,240 (this is the largest one that fits in our gpu memory). ## conclusion and future work in this work, we have shown that insights from computer vision on the regularizing effect of small batch sizes are also applicable to nmt. our results, focused on a low-resource setting, challenge those of previous nmt studies with large amounts of training data, and the general belief that batch sizes should be as large as they fit in the gpu memory. we have shown that training with small batch sizes leads to models that generalize better, and found the optimal batch size below which performance degrades. future work should explore how the learning rate must be adjusted depending on the batch size, and whether a dynamically scheduled combination of batch size and learning rate can provide an even better regularizer. for instance, it should be tested if dynamic batch sizes as proposed by @xcite can also improve performance in a low-resource setting, with batch size thresholds changed to measure an optimal level of noise.