Measuring the Impact of Data Augmentation Methods for Extremely Low-Resource NMT (2023)

Abstract: Data augmentation (DA) is a popular strategy to boost performance on neural machine translation tasks. The impact of data augmentation in low-resource environments, particularly for diverse and scarce languages, is understudied. In this paper, we introduce a simple yet novel metric to measure the impact of several different data augmentation strategies. This metric, which we call Data Augmentation Advantage (DAA), quantifies how many true data pairs a synthetic data pair is worth in a particular experimental context. We demonstrate the utility of this metric by training models for several linguistically-varied datasets using the data augmentation methods of back-translation, SwitchOut, and sentence concatenation. In lower-resource tasks, DAA is an especially valuable metric for comparing DA performance as it provides a more effective way to quantify gains when BLEU scores are especially small and results across diverse languages are more divergent and difficult to assess.

## introduction neural machine translation (nmt) has been established as the dominant approach for developing state-of-the-art machine translation (mt) systems. the neural network-based architecture enables effective translation without expert linguistic knowledge while better capturing contextual information. however, many nmt systems are data-inefficient and are dependent on large amounts of parallel data pairs in order to attain reliable performance, limiting their applicability in low-resource tasks. this paper is particularly interested in applicability of da methods in the preservation of low-resource and scarce languages. there is therefore a significant performance gap in nmt for low-resource language pairs @xcite . one way that this gap is addressed is the generation of synthetic data through unsupervised data augmentation (da). da has been largely used in other deep learning modalities like image-and tabular-based data @xcite . multilingual text da, in particular, has been the frontier of da research. @xcite proposed the backtranslation of sentences from monolingual data to generate bitext for a pseudoparallel corpora. recently, many more new da approaches have been presented in order to improve nmt systems. as opposed to approaches for low-resource nmt exploiting auxiliary languages through transfer learning, which rely heavily on the availability of data on a rich-resourced and linguistically similar language, da in particular has potential to expand language technologies further by addressing that many low-resource and indigenous languages tend to be the most specialized and idiosyncratic, and are often part of smaller language families that are endangered as a whole @xcite ). da thus is especially relevant in the preservation of low-resource and scarce languages. however, da methods often do not exhibit consistent improvement across translation tasks @xcite . in the case of low-resource languages, the effectiveness of da may be even more irregular. synthetic pairs based on very limited amounts of data may have compromised quality, and the generalizability of these methods for scarce and orthographically diverse languages is understudied. in this paper, we propose a method to measure the impact of da on machine translation tasks in a low-resource environment. we then use this metric to assess the performance of three da methodsback translation, switch-out, and sentence concatenation-on a machine translation task. we first measure the impact of da on variously-sized subsets of high-resource language datasets, including english-italian, english-turkish, and german-english, to assess the generalizability and consistency of the selected da methods. we then demonstrate how da methods can be employed and measured in truly scarce linguistic environments by measuring the impact of da on a machine translation task for the language pairs english-romany, english-māori, english-uyghur, and english-kabyle. ## background we implement and investigate three multilingual da approaches in our analyses. each of these approaches have been shown to improve performance in high-resource environments, underscoring the importance of measuring the impact of such approaches in low-resource settings as well. ## datasets in this paper, we use two groups of data. both groups of data are from the tatoeba dataset @xcite . the training data for the tatoeba dataset was obtained from opus' parallel corpora @xcite , which is made up of translated texts from the web. first, we determine the generalizability and consistency of different da algorithms by measuring their performance in a simulated low-resource environment, that is, using small samples of high-resource languages. second, we measure the efficacy of da algorithms for true low-resource environments using scarce language datasets. to simulate a low-resource setting we randomly sample 1m pairs each from the english-italian, english-turkish, and german-english tatoeba dataset training data. we train multiple models by sampling the data in increments of 5%, 10%, 20%, 50%, and 100%. we sample from the unused portions of the dataset for use in the augmentation methods requiring monolingual data. we report results on the 2021 test sets. second, we test da methods for four truly scarce language pairs, including english-romany (24k pairs), english- māori (221k pairs), english-uyghur (143k pairs), and english-kabyle (84k pairs) (see section 2.2 above). we train multiple models by sampling the data in increments of 25%, 50%, 75%, and 100%. ## data augmentation advantage across da methods, synthetic pairs contribute different amounts of value to the training data. in some cases, synthetic pairs have the same impact as a true pair in the training pair, while in other cases, synthetic pairs seem to have no value or even negative value within the training dataset. in this section, we offer a simple yet novel metric to quantify how many true data pairs a synthetic data pair is worth. we call this metric data augmentation advantage (daa). we calculate daa as follows. first, we perform linear interpolation for the baseline model, where x is the number of training pairs and y is a bleu score. then for a point y we calculate the interpolant as in equation 1 below. note that 𝑦𝑦 𝑎𝑎 < 𝑦𝑦 < 𝑦𝑦 𝑏𝑏 and 𝑥𝑥 𝑎𝑎 < 𝑥𝑥 < 𝑥𝑥 𝑏𝑏 . for a specified target bleu score b on the linear interpolation described above, let xt be the number of training pairs needed to achieve b in the current experiment, and let xb be the number of training pairs needed to achieve b in the baseline model. we can then calculate xadv as in equation 1 below. using xadv, we calculate data augmentation advantage (daa) as in equation 3 . this process is summarized in figure 1 . daa represents the number of true data points that each synthetic data point is worth. for example, if daa is 0.5, then the addition of da is comparable to having 50% more data and a synthetic data point is worth 0.5 true data points. in the results section below, the overall daa values of a da method are obtained by averaging the values across the language tasks. ## experiments for all the experiments, we use the opennmt-py toolkit @xcite for the translation models. the nmt system is a 4-layer attention-based encoder-decoder model @xcite . this system estimates the probability distribution of a sentence in the target languages given a sentence in the source language. an encoder recurrent neural network (rnn) maps each source word to a word vector; the word vectors are then mapped to a set of hidden vectors. the decoder rnn decodes the source-side hidden vectors to predict the next word in the target languages. note that target-side decoder used is aware of the previously generated words. we train the model with hidden dimension 1024 and batch size 64. we use a dropout probability of 0.3. we employ early stopping to train until convergence in order to control for the role of training time in performance changes. the settings used in training the models are the same for each language pair. across the experiments we process the source and target language sentences with byte-pair encoding (bpe) @xcite based on the sentencepiece subword model @xcite with a vocabulary size of 8k. bpe is a method for segmenting words into subword units based on their frequency of occurrence. it enables better coverage and generalization in handling rare and out-ofvocabulary words by breaking them down, and is especially relevant in languages with complex morphology such as turkish. sentencepiece is a powerful and flexible method for unsupervised tokenization and subword segmentation, and provides an implementation of the bpe algorithm. for models with augmentation, bpe is applied after da, and for augmentation methods with an intermediate model, bpe is applied for each. for all da methods, we generate synthetic data with a 1:1 ratio. in the experiments, we compare a baseline model with no augmentation to models trained with the original training data in addition to the synthetic data obtained through each da method. we additionally ran control experiments by simply duplicating the data to verify that any results were due to da, and observed no improvement from the baselines. we run a model for each of the subsets of data and report our final results for each. the translation quality is measured by a single reference bleu score @xcite . three language pairs, english-italian, english-turkish, and german-english, are used to assess the generalizability and consistency of the methods. ## conclusion & future work data augmentation (da) is a popular strategy to boost performance on neural machine translation tasks. the impact of data augmentation in lowresource environments, particularly for diverse and scarce languages, is understudied. in this paper, we introduce a simple yet novel metric to measure the impact of several different data augmentation strategies. this metric, which we call data augmentation advantage (daa), quantifies how many true data pairs a synthetic data pair is worth in a particular experimental context. because daa provides a consistent measure comparable across the results, we are able to determine that switchout and sentence concatenation show the greatest language task generalizability, providing more consistent daa. in general, switchout is especially advantageous with less available data, most evident in the increasing daa in @xmath0, while back-translation has more limitations with regards to the minimum amount of data required for best performance. in particular, in lower-resource tasks, daa is an especially valuable metric for comparing da performance as it provides a more effective way to quantify gains when bleu scores are especially small and results across diverse languages are more divergent and difficult to assess.