Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach (2021)

Abstract: In the context of neural machine translation, data augmentation (DA) techniques may be used for generating additional training samples when the available parallel data are scarce. Many DA approaches aim at expanding the support of the empirical data distribution by generating new sentence pairs that contain infrequent words, thus making it closer to the true data distribution of parallel sentences. In this paper, we propose to follow a completely different approach and present a multi-task DA approach in which we generate new sentence pairs with transformations, such as reversing the order of the target sentence, which produce unfluent target sentences. During training, these augmented sentences are used as auxiliary tasks in a multi-task framework with the aim of providing new contexts where the target prefix is not informative enough to predict the next word. This strengthens the encoder and forces the decoder to pay more attention to the source representations of the encoder. Experiments carried out on six low-resource translation tasks show consistent improvements over the baseline and over DA methods aiming at extending the support of the empirical data distribution. The systems trained with our approach rely more on the source tokens, are more robust against domain shift and suffer less hallucinations.

## introduction in order to train reliable neural machine translation (nmt) systems, large amounts of parallel sentences -sentence pairs in two languages that are mutual translations-are needed, which constitutes a critical barrier for low-resource language pairs. this problem has been addressed through different approaches, such as transfer-learning from highresource language pairs @xcite , using linguistic annotations @xcite , training multilingual systems @xcite and applying data augmentation strategies @xcite , i.e., artificially generating additional parallel sentences. data augmentation (da) is formalised by many authors as a solution to a data distribution mismatch problem @xcite . the data distribution of the sentence pairs observed in the training corpus, p(x, y), differs from the true data distribution, p(x, y). hence, the system should be trained on a training set that follows q(x, y), an augmented version of p(x, y) with a wider support. in this way, the trained system is less likely to face totally out-of-distribution data when translating. in this paper, we propose a completely different framework for da in which we generate additional parallel sentences which, despite being completely unlikely under the data distribution, systematically improve the quality of the resulting nmt system. inspired by one-to-many multilingual nmt, where richer encoder representations are obtained @xcite , we propose a set of simple da strategies to produce synthetic target sentences aimed at strengthening the encoder. these strategies expose the network during training to new situations where the target-language context is not sufficient to achieve a low loss, and the burden is passed on to the encoder. recent findings by @xcite further motivate our approach: they claim that the influence of source tokens in the output predictions of an nmt system decreases as decoding advances. moreover, to avoid harmful interferences by the out-of-distribution target data generated, we follow a simple multi-task learning (mtl) approach that does not require changes to the model architecture. we call the proposed framework multi-task learning data augmentation (mtl da) to stress the fact that the augmented data, which do not follow the distribution of parallel sentences in the training corpus, constitute different auxiliary tasks that nevertheless produce a positive transfer to the main task. our framework does not require elaborated preprocessing steps, training additional systems, or data besides the available training parallel corpora. experiments with six low-resource translation tasks show that it systematically outperforms two powerful methods aiming at extending the support of the empirical data distribution -raml @xcite and switchout @xcite and that it can be combined with synthetic corpora generated through back-translation @xcite to get further improvements. in the context of explainable deep learning models, we perform an analysis of the relevance of the encoder and decoder representations in the nmt system output, which shows that, thanks to the auxiliary tasks, mtl da increases the contribution of the source tokens to the decisions made by the nmt system. moreover, systems trained with mtl da are much more robust against domain shift and produce less hallucinations @xcite . the remainder of the paper is organised as follows. next section briefly describes our mtl da approach and the different auxiliary tasks we evaluated. after that, sec. 3 describes the experimental settings, whereas sec. 4 provides and discusses the results obtained. sec. 5 then presents an analysis of the changes in the transformer dynamics induced by our auxiliary tasks as a way of explaining the improvement in translation quality. the paper ends with a review of the most relevant works in the area of da for nmt in sec. 6, followed by some concluding remarks in sec. 7. ## multi-task learning approach and auxiliary tasks we propose a simple mtl approach that consists of using a vanilla nmt system -in our experiments, it is a transformer system as defined by @xcite -where all (main and auxiliary) tasks share the encoder and the decoder. in order to avoid harmful interferences by the out-ofdistribution target data generated for the auxiliary tasks, we add a task-specific artificial token to the source sentence to constrain the kind of output to be produced @xcite , much like in multilingual nmt. for each auxiliary task, we append a synthetic corpus of the same size to the original training data, which is obtained by applying a transformation to each original pair of sentences. in almost all the tasks, the source sentence is left unchanged while the target sentence is substantially modified. what follows is a brief explanation of the transformations we have tested and their expected effect on the training dynamics of the encoder. some of them have been previously applied in da, but never in an mtl set-up such as the one we are presenting. some transformations are controlled by a hyperparameter α that determines the proportion of target words affected by the transformation. in what follows, t denotes the amount of words in the original target sentence. task lang. synthetic training sample source es gibt andere möglichkeiten , die pyramide zu durchbrechen . original training sample target there 's other ways of breaking the pyramid . swap target there . other ways of breaking pyramid 's the token target there 's other unk of unk unk unk . source target es gibt andere möglichkeiten , die pyramide zu durchbrechen . reverse target . pyramid the breaking of ways other 's there mono target 's there other ways the pyramid of breaking . replace source es gibt aufzurüsten kalt , schach spezialwissen zu durchbrechen . target there 's arming cold of breaking chess specialties . ## experimental settings we have conducted experiments for the translation from english to german, hebrew and vietnamese, and for the translation in the reverse direction, using corpora commonly used for evaluating da techniques in low-resource scenarios. we evaluated the effect of using each of the mtl da auxiliary tasks, as well as the combination of the best performing ones. we also evaluated two strong da methods that aim at extending the support of the empirical data distribution by replacing some words by random samples from the vocabulary: switchout (applied on the source side), raml (applied on the target side), and the combination of both. datasets. following @xcite and @xcite , for english-german and english-hebrew we used the training data (speeches of ted and tedx talks) of the iwslt 2014 text translation track @xcite ); foot_1 for development and testing we used the tst2013 and tst2014 datasets, respectively. like @xcite , for english-vietnamese we used the training data (also ted talks) of the iwslt 2015 text translation track @xcite ; foot_2 datasets tst2012 and tst2013 were used, respectively, for development and testing. to evaluate the impact of mtl da when it is combined with synthetic data obtained through back-translation @xcite -that is, by translating a target-language monolingual corpus into the source language-, we collected additional english monolingual data, back-translated it into german, hebrew and vietnamese, and added the resulting synthetic corpus to the training data listed above. the monolingual data used consists of all the available monolingual english sentences in the iwslt 2018 shared task on low-resource mt of ted talks that were not present in the parallel training data described above. one-to-many word alignments in both translation directions were obtained using mgiza++ @xcite . 6 source-to-target word alignments were used for the mono transformations; the one-toone word alignments required by the replace transformation were obtained by computing the intersection between the one-to-many word alignments in both translation directions. the bilingual lexicon for the replace transformation was built by associating to each source word the target word it is most frequently aligned with in the one-to-one word alignments. training. our neural model is a transformerbased model as defined by @xcite , with the exception of the amount of warmup_steps, which was set to 8,000. all the experiments were carried out on a single gpu with mini-batches made of 4,000 tokens. validation was done every 1,000 updates, and the patience based on the bleu score on the development set was set to 6 validation cycles; we then kept the intermediate model performing best on the development set. we trained the systems with the fairseq toolkit @xcite . for raml and switchout, we integrated into fairseq the sampling function released by @xcite . 7 systems trained with mtl da were fine-tuned on the main (translation) task after being trained on the combination of the main and auxiliary tasks. when combining different auxiliary tasks, a different special token was used for each one. da hyperparameters. the proportion of words affected by the swap, token and replace transformations is controlled by a hyperparameter α, whereas raml and switchout are governed by a temperature τ . for each language pair, we explored values of α in [0.1, 0.9] at intervals of 0.1, and values of τ around the best values reported by @xcite . 8 the results reported are those obtained with the model that maximizes bleu on the development set. the best hyperparameters obtained for the experiments with the iwslt parallel data were reused for the experiments with the training set extended with back-translated data. ## results and discussion iwslt parallel data. iwslt parallel data + back-translated data task de-en he-en vi-en baseline 31.3 ± 0.5 34.5 ± 0.1 29.3 ± 0.3 switchout+raml 31.7 ± 0.8 34.1 ± 0.7 29.9 ± 0.5 reverse+mono+replace 32.3 ± 0.1 35.3 ± 0.3 30.4 ± 0.7 interestingly, using the three best auxiliary tasks together further improves the performance, achieving the best results in all translation tasks with bleu scores between 1.1 and 1.9 points over the baseline. 10 this suggests that different auxiliary tasks affect the encoder in different ways and are somehow complementary. a comparison of our mtl da approach with raml, switchout and their combination (switchout+raml) shows that our approach, being much simpler in nature, also outperforms them. back-translated data. ## explainability to confirm that the systematic improvements in translation quality and enhanced domain robustness are related to the encoder being exposed during training to more situations where a good source representation is crucial, we carried out an analysis of the relative source and target contributions to the generation decisions of the nmt system. according to @xcite , systems trained with more data tend to rely more on source information; we expect mtl da to produce the same effect. another aspect that will account for the positive impact of mtl da in the system's encoder is the generation of hallucinations @xcite : completely inadequate translations that usually occur under domain shift @xcite , due to the system relying too much on the target context @xcite . we expect systems trained with mtl da to produce less hallucinations. to validate this last hypothesis, we carried out an hallucination analysis on the results of the domain shift experiments. relative source and target contributions. we used layer-wise relevance propagation (lrp), as adapted to transformers by @xcite , to compute the relative contribution of source and target tokens to each prediction made by the system. lrp allows us to compute r t (x i ) and r t (y j ), the relative contribution of source token x i and target token y j , respectively, to the prediction emitted by the network at time t. the total relevance at each time step is 1, i.e. for all time steps t the following equation holds, where r t (x) and r t (y) stand for the total contribution of the source tokens and target tokens, respectively: to have reliable comparisons, we also follow @xcite and evaluate the relative source and target contributions on a subset of sentences from a held-out corpus with the same source length and the same target length. in this way, we can fairly compare different strategies, since we teacher-force the reference translations when computing lrp so as to obtain translations with exactly the same length. the held-out corpus used is the concatenation of all the development corpora released for the corresponding iwslt task, foot_10 while the subset chosen is the largest set of parallel sentences with the same source length and the same target length, as long as there are at least 16 tokens in each side. 12 to compute the relative contributions, we retrained the baseline and the mtl da systems featuring the best performing auxiliary tasks on the iwslt parallel data with the toolkit released by @xcite . 13 figure 1 depicts the total source contribution at each generation time step t for the different translation tasks and systems. we skip the first time step, when there is no target prefix available, and show the source contribution for the eos token too. for mtl da models, the source contains the special token corresponding to the main task. for the translation tasks with english as source language, we can observe the same trends as @xcite : source influence decreases as decoding progresses. there is also a peak in the penultito the production of the target tokens. best2 stands for the combination of the two best auxiliary tasks within our mtl da framework: reverse and replace. best3 stands for the combination of the three best ones: reverse, replace and mono. finally, no consistent differences in source influence could be found between the reverse and replace auxiliary tasks. the systems combining multiple auxiliary tasks, however, are consistently the ones with the highest source influence, thus confirming the complementarity of the tasks. hallucinations. to estimate the number of hallucinations produced by the systems evaluated, we follow the procedure proposed by @xcite and used by @xcite . although their interest was in detecting those sentences that induced the generation of hallucinations after introducing spurious tokens in the input, we adapted it to automatically measure the number of input sentences in a test set for which the corresponding output seems to be an hallucination. to this end, we use an adjusted version of bleu which only takes into account the precision of unigrams and bigrams with weights 0.8 and 0.2, respectively, as proposed by @xcite . if the sentence-level adjusted bleu of the lowercased emitted translation is below a certain threshold (10 in our experiments), it is taken as a sign of hallucination. we evaluate the tendency to produce hallucination of the baseline as compared to our mtl da approach combining the auxiliary tasks reverse, mono and replace, and to the switchout, raml and switchout+raml systems. sentences whose translations cannot be regarded as hallucinations are not relevant to our study, neither are those for which the baseline and the system to which it is compared to both hallucinate. we therefore count the number of sentences that induce an hallucination on one of the systems but not on the other. 3 . we consider that the other system is not hallucinating if its adjusted bleu is at least 20 bleu points higher. fig. 2 represents these data for the english-german translation tasks for the same domains (indomain, it, legal and medical) and corpora used for the domain robustness evaluation reported in sec. 4. the grey bar represents the number of hallucinations of the baseline that are not labeled as such by the corresponding da system; consequently, higher values are better. additionally, the color bars represent the number of hallucinations of the da system that cannot be labeled as such in the baseline system (shorter bars are better). as can be seen, disjoint hallucinations barely happen with in-domain data, but they can be easily found when considering out-of-domain data. in every domain and in both translation directions, the blue bar (the one representing our mtl da method) is clearly the shortest one in almost all cases, while at the same time the grey bar is the largest one. this is a clear sign of reduction of hallucinations in the systems trained with mtl da. appendix c presents supplementary details on the bleu thresholds and the smoothing technique considered when computing the scores. ## related work the back-translation @xcite approach for leveraging additional target monolingual data to produce additional training samples is probably the most popular da approach for nmt. the set of related approaches covered in this section, however, mainly focus on methods that, as mtl da, do not require additional resources besides the training parallel corpus. @xcite evaluate back-and forwardtranslation in such a setting. they train forward and backward nmt systems on the available parallel data and use them to produce new synthetic samples by translating either the target side @xcite or the source side @xcite of the original training corpus. the approaches we have evaluated in our experiments, raml @xcite and its extension to the source language, switchout @xcite , aim at extending the support of the empirical data distribution and keeping it smooth (similar sentence pairs have similar probabilities). to that end, they replace words with other words sampled from a uniform distribution over the vocabulary, which, in practice, results in infrequent words being overrepresented. @xcite presented a related approach to encourage compositional behaviour: replaced words are selected from another sentence and not from the vocabulary. some of our auxiliary tasks have already been used for da, but mostly on the source side and rarely in an mtl framework. replacing tokens with placeholders (as we do in token) has already been applied by @xcite to the source language, in combination with auxiliary tasks involving detecting replaced and dropped tokens. @xcite also evaluate the impact of replacements on the target data, but do not follow an mtl approach. word dropout @xcite @xcite can also be considered a related approach. regarding changes to word order, in addition to the proposals by @xcite and @xcite , it is worth highlighting the strategy proposed by @xcite who apply a selftranslation approach using a right-to-left decoder. unlike our mtl da framework, they need to generate translations from the model during training and adjust multiple terms in the training loss. there are more da approaches based on replacing words which are worth mentioning. @xcite randomly replace words in the source side of the training samples. @xcite replace words selected at random with soft words whose representations are obtained from the probability distribution provided by a language model. @xcite replace a number of words in their training samples by infrequent words in order to improve the performance of the nmt model when dealing with them at translation time. words to be replaced are identified using a large source language model. once the source words to be replaced are identified, a word-alignment model and a probabilistic dictionary are used to also replace the corresponding counterpart by the most probable translation of the new source word. in our mtl da framework, the replace transformation, which is similar to @xcite 's work, does not require any language model. regarding back-translation, edunov et al. (2018) apply several simple transformations (word deletion, replacement, swapping) to back-translated data reporting a noticeable improvement. in relation with the special token we use to prevent negative transfer between tasks, @xcite propose a similar strategy to identify synthetic samples when combining actual parallel data and backtranslated data for training. @xcite extends this work by including forward-translated data for training using two different special tokens to distinguish the two types of synthetic data. ## concluding remarks in this paper, we have presented a multi-task learning approach for data augmentation (mtl da) in nmt. we deviate from common approaches that aim at extending the support of the empirical data distribution by generating new samples that are likely under such distribution. we propose instead to carry out da in a mtl manner, by artificially generating new sentence pairs with aggressive transformations, such as reversing the order of the target sentence, which may make the target sentence completely unfluent. translating into these augmented sentences constitute new tasks that provide new contexts during training where the target prefix is not informative enough to predict the next word, thus strengthening the encoder and forcing the system to rely more on it. experiments carried out on six low-resource translation tasks that usually serve as benchmark for da show consistent improvements over a base-line system (on average around 1.6 bleu points) and over strong da methods that aim at extending the support of the empirical data distribution without mtl. moreover, additional analyses show that the systems trained with mtl da rely more on the source tokens, are more robust against domain shift and suffer less hallucinations. mtl da is agnostic to the nmt model architecture and does not require elaborated preprocessing steps, training additional systems, or data besides the available training parallel corpora. furthermore, it could be combined with existing da methods, in addition to back-translation, specially those that operate on the source side @xcite , since our transformations mainly address the target. we expect this strategy to inspire the implementation of new auxiliary tasks to be used for mtl da, specially those aimed at improving the training dynamics of the system. we believe that further improvements could be obtained by following more elaborated strategies for multi-task learning, such as changing the proportion of data for the different tasks, evaluating different ways of parameter sharing between the different tasks (e.g. sharing the encoder but not the decoder), and using other training schedules @xcite . finally, we conclude that making the encoder representation essential to minimize the loss during training should be embraced as a potential way of boosting nmt quality.