Targeted Data Augmentation Improves Context-aware Neural Machine Translation (2023)

Abstract: Progress in document-level Machine Translation is hindered by the lack of parallel training data that include context information. In this work, we evaluate the potential of data augmentation techniques to circumvent these limitations, showing that significant gains can be achieved via upsampling, similar context sampling and back-translations, targeted on context-relevant data. We apply these methods on standard document-level datasets in English-German and English-French and demonstrate their relevance to improve the translation of contextual phenomena. In particular, we show that relatively small volumes of targeted data augmentation lead to significant improvements over a strong context-concatenation baseline and standard back-translation of document-level data. We also compare the accuracy of the selected methods depending on data volumes or distance to relevant context information, and explore their use in combination.

## related work a variety of studies have tackled context-aware approaches within the framework of nmt, analysing the improvements that these models can provide over non-contextual baselines @xcite @xcite @xcite . one of the first methods proposed for the task is the concatenation of context sentences to the sentence to be translated @xcite . this simple approach is still one of the most efficient methods to perform context-aware neural machine translation, matching or outperforming more sophisticated ones @xcite . alternative methods have involved refining the context-agnostic translations @xcite @xcite , or modelling context information with specific nmt architectures @xcite @xcite @xcite . the growing interest in context-aware nmt models has increased the need for parallel data where context information is preserved. dedicated efforts have been made to increase the availability of this type of data, for instance in recent shared tasks in the wmt series @xcite . however, context boundaries might not always be recoverable, ensuring continuous contextual information in sentence-aligned datasets can be a costly task, and most of the available relevant data might be limited to specific domains. data augmentation might thus complement the existing datasets for the variety of possible language pairs and domains. over the years, specific efforts have been made to create synthetic data to improve nmt at the sentence-level @xcite @xcite @xcite . the most widespread method is the use of back-translations, a technique introduced to nmt by @xcite that exploits monolingual corpora by machinetranslating target language data into the source language. for document-level nmt, backtranslation has been shown to be effective in capturing contextual information, both by translating the original data sentence by sentence (junczys-dowmunt, 2019) or by using contextaware models @xcite . in the same vein, @xcite find that document-level models benefit even more from back-translations than their sentence-level counterparts. to our knowledge, back-translations targeted on specific phenomena, as proposed by @xcite for sentence-level models, have not been investigated for contextaware nmt and we include this approach among our data augmentation methods. monolingual data have also been exploited for document-level nmt via context-level decoders @xcite or systems that learn to improve the translations generated by sentence-level models @xcite . other methods augment document-level parallel data by creating synthetic sentence sequences via the concatenation of varying numbers of sentences extracted from aligned document pairs @xcite @xcite . other forms of data augmentation are antecedent-free augmentation @xcite , which creates new training examples by modifying cases where the antecedent is not present in the available context, or the more recent method of @xcite , which generates faulty data and trains nmt models via contrastive learning. in both cases, a coreference analysis needs to be performed on document pairs. finally, data augmentation has also been performed for sentence-level models by mining large volumes of comparable data @xcite . this type of data has been shown to increase the quality of nmt models for low-resource languages, independently or in combination with back-translations @xcite . to our knowledge, using similar data for contextual data augmentation has not yet been explored, and we include a variant of this method in our analysis. context-aware models are particularly suited to improve the translation of phenomena that directly depend on context information, such as intersentential anaphora resolution, discourse coherence or terminological consistency @xcite . we evaluate our approach on the specific task of adequately translating pronouns in context, for which several specific test sets have been created @xcite @xcite @xcite . ## methodology we aim to generate synthetic parallel data that include relevant information for the translation of specific contextual phenomena. this involves (i) identifying context blocks in document-level data, i.e. parallel sequences consisting of a sentence and its previous context sentence in the source and target languages, and (ii) sampling blocks that contain elements whose translation typically requires context information. although our approach could be applied to other contextual phenomena as well, we selected pronouns as our linguistic category of interest, specifically the translation of pronouns from english into german and french, given their relevance for document-level translation and the availability of contrastive test sets for precise evaluations @xcite . in particular, for english-german, we focused on the pronoun it which can be translated as es (neutral gender), er (masculine) or sie (feminine). for english-french, in addition to it, which can be translated as elle (feminine) or il (masculine), we also included they, which can be translated as elles (feminine) or ils (masculine). we first identify context blocks where the targeted elements occur in the source (src i ) and target (tgt i ) sentences and the preceding source sentence (src i-1 ) is available. more specifically, we extracted context blocks that met one of the following conditions: (i) it in src i and es/er/sie in tgt i (en-de) (ii) it in src i and elle/il in tgt i (en-fr) (iii) they in src i and elles/ils in tgt i (en-fr). under this approach, we might sample data where the antecedent of the pronoun is found in the block, but might also extract blocks where the antecedent is not included. these instances can also be useful as they might help balance the data in case of bias. this extraction method avoids having to use coreference annotation tools, which simplifies the data extraction process. to avoid introducing ambiguity in the sampled data, we discarded cases where more than one pronominal translation with different genders appeared in the target sentence. after sampling the blocks of interest, we create new ones by either duplicating the sampled blocks (upsampling), replacing the context sentences randomly or via sentence embedding similarity (context sampling), or back-translating the target language blocks (targeted backtranslation). we describe each method in more details below. upsampling. this method (hereafter, up-samp) is the simplest, and consists in repeating the selected blocks multiple times and adding them to the training data. this type of data augmentation could lead to overfitting, i.e. overtraining the model on the upsampled data and learning specific patterns which might be irrelevant in other cases. it may thus happen that the model achieves higher accuracy on the selected data but does not generalise well to other data. ## optimal variants we first aimed to establish the optimal selection of data along two lines: (i) balancing the distribution of pronominal categories vs. maintaining an unbalanced distribution, and (ii) varying the amounts of sampled data with each method. ## method comparison in this section, we compare the methods selected in the previous section, i.e. balanced 221,327 for english-german and balanced 15k for english-french, except for english-french t-bt, with 25k selected. additionally, for each language pair we trained a combined model (comb) where we merged the augmentation blocks from each method and selected a random sample maintaining distribution balance. we discarded the option of using the combination of all data, as this would have resulted in unbalanced data distributions. ## conclusions in this work, we described three different data augmentation techniques for context-aware nmt and evaluated them in isolation and in combination over standard sentence-level and documentlevel test sets. specifically, we created synthetic data centred on improving pronoun translation in english-german and english-french, as a test case for an approach which could be applied to other contextual phenomena as well, provided they feature overt elements that may be targeted. the methods we examined included upsampling, context sampling with both random and similar context substitution, and back-translations, all targeted on specific data featuring different pronominal types. all methods improved over a strong concatenation baseline, in terms of accuracy on contrastive test sets, while also achieving parity or improving in terms of bleu scores in most cases. accuracy improvements were markedly larger on the english-german contrastive sets, as high scores could already be obtained by the baseline on the english-french test sets. we leave for future work an exploration of alternative contrastive datasets and models with a wider contextual window. we demonstrated that balancing the data and using minimal volumes was optimal overall, and showed that the improvements were mainly obtained by leveraging contextual information in preceding sentences. all methods were shown to perform markedly better than simply back-translating document-level data, indicating that targeted data augmentation might be a research path worth exploring further for context-aware nmt. finally, among the selected methods, targeted back-translation proved a simple and effective approach which performed well across the board, although it can be outperformed in terms of accuracy on specific categories. this method does not require external tools such as coreference resolvers and can significantly improve the results of a 2to1 model with relatively small amounts of data, as measured in contrastive evaluations as well as evaluations in terms of bleu. the combination of data from the different examined methods may also be considered a viable alternative, as it resulted in balanced improvements over categories overall.