Promoting Target Data in Context-aware Neural Machine Translation (2024)

Abstract: Standard context-aware neural machine translation (NMT) typically relies on parallel document-level data, exploiting both source and target contexts. Concatenation-based approaches in particular, still a strong baseline for document-level NMT, prepend source and/or target context sentences to the sentences to be translated, with model variants that exploit equal amounts of source and target data on each side achieving state-of-the-art results. In this work, we investigate whether target data should be further promoted within standard concatenation-based approaches, as most document-level phenomena rely on information that is present on the target language side. We evaluate novel concatenation-based variants where the target context is prepended to the source language, either in isolation or in combination with the source context. Experimental results in English-Russian and Basque-Spanish show that including target context in the source leads to large improvements on target language phenomena. On source-dependent phenomena, using only target language context in the source achieves parity with state-of-the-art concatenation approaches, or slightly underperforms, whereas combining source and target context on the source side leads to significant gains across the board.

## introduction significant progress has been achieved in machine translation within the neural machine translation (nmt) paradigm @xcite @xcite . for the most part though, most nmt models translate sentences in isolation, preventing the adequate translation on document-level phenomena such as cohesion, discourse coherence or intersentential anaphora resolution @xcite @xcite @xcite . among the various approaches to context-aware nmt, simple concatenation of context sentences, as initially proposed by @xcite , remains a solid baseline typically used in practice with varying amounts of source-target context pairs @xcite @xcite @xcite . context-aware models typically rely on parallel document-level data, a scarce resource overall despite recent efforts to provide this type of resource @xcite @xcite . to the exception of approaches such as the monolingual repair framework of @xcite , context data in the source language is generally used as the core information to model context-awareness. however, most discourse-level phenomena feature information that is either present mainly in the target language (e.g., lexical cohesion, deixis) or in both the source and target languages (e.g., gender selection, ellipsis). considering this, in this work we aim to explore the impact of promoting target language data in standard context-aware nmt. along these lines, we explore a simple concatenation-based approach which consists in simply prepending context sentences from the target language to the source sentence to be translated, in isolation or in combination with source context. the underlying intuition is that contextual phenomena would be mainly modelled at the decoder level via target-side context information, whereas, on the encoder side, context data will be either ignored and copied, as foreign data, or also associated with source information to further model context. using target language context data on the source side also enables the use of a standard nmt architecture and concatenation-based approach to context-aware nmt. we show that replacing source context sentences with the target context already leads to significant gains for discourse-level phenomena that depend on target-language information, while achieving either parity or moderate degradation in contrastive accuracy on other phenomena. combining both source and target context sentences on the source side leads to consistent significant improvements across the board. we establish our results on two language pairs, english-russian and basque-spanish, for which contrastive test sets are publicly available on a range of phenomena that depend on the source and/or target language context. in addition to accuracy results on specific phenomena, we compare the overall translation quality on parallel test sets as well. we also measure the impact of using either reference or machinetranslated output as context at inference time, with only minor loss observed with the latter in our experiments. finally, we evaluate the use of backtranslated data, with similar comparative gains as those obtained using parallel document-level data. overall, our experimental results indicate that promoting target context data within a standard nmt architecture can be a promising alternative for context-aware machine translation. ## related work one of the first methods proposed for documentlevel nmt is the concatenation of context sentences to the sentence to be translated, in either the source language only, or in both source and target languages @xcite . this method does not require any architectural change and uses a fixed contextual window of sentences. it provides a robust baseline that often achieves performances comparable to that of more sophisticated methods, in particular in high-resource scenarios @xcite @xcite . variants of this approach include discounting the loss generated by the context @xcite , extending model capacity @xcite or encoding the specific position of the context sentences @xcite . alternative approaches include refining contextagnostic translations @xcite and modelling context information with specific nmt architectures @xcite @xcite . more recently, the use of pretrained language models has been explored for the task, using them to encode the context @xcite , to initialize nmt models @xcite or fusing the language model with a sentence-level translation model @xcite . directly using pretrained language models to perform translation can achieve competitive results, although these models might still produce critical errors and sometimes perform worse than conventional nmt models @xcite @xcite . concatenation-based approaches vary regarding their use of context, exploiting either the source context @xcite , the target context @xcite or both @xcite @xcite . the benefits of using context sentences in both the source and the target languages are also discussed in @xcite , for a multi-encoder approach. @xcite conclude that concatenation-based models make more use of the target context than the source context, but @xcite show that the effectiveness of the target context versus the source context is highly dependent on the language pair involved. close to the target-based approach we explore in this work, @xcite and @xcite include variants where target data is concatenated to the source sentence, notably showing that the target context is equally as important than source context, and particularly beneficial to address target-level phenomena. however, their experiments were limited to one target sentence, i.e. without prepending context on the target side. we show in this work that including the target con- @xcite and @xcite . since standard nmt evaluation metrics such as bleu @xcite are not well equipped to assess accuracy on discourse phenomena, several challenge test sets have been developed specifically to measure translations in context, via contrastive evaluations @xcite @xcite @xcite @xcite . we include contrastive test sets that cover target-language phenomena such as deixis or lexical cohesion, as well as phenomena where the relevant context information is available in both source and target languages. ## exploiting target language context the main incentive for the promotion of target context data is the nature of the contextual phenomena of interest for machine translation, as these can be grouped into four broad categories depending on the location of the relevant contextual information. in a first category would be discourse-level phenomena that require context information on the target language side, typically related to discursive cohesion in a broad sense (see examples a and b in in a second major category are phenomena for which the relevant context information is in both the source and the target context (examples c and d in finally, a fourth broad category contains constructions where the source and target context need to be processed in combination for a correct translation. in the example b in any target-only approach, such as monolingual repair @xcite or the targetonly variant we also explore in this work, would only generate the correct translation in the latter two classes of cases by either chance or training bias. although these cases exist, it is unclear how widespread they actually are, compared to the other two main classes of contextual phenomena described above. in what follows, we set to compare the relative importance of source and target data across the main phenomena as represented in the selected document-level test suites. ## promoting target language data to explore the promotion of target language data, we simply prepend the target context sentences to the source sentence to be translated, either discarding or maintaining the source context sentences. on the target side, we evaluate the use of empty context as well as maintaining the target context sentences. we add a special token to separate the concatenated context sentences in all cases. at inference time, in practice the previously translated sentences would be prepended as context. since context translations can feature various degrees of correctness, we assess the approach under both ideal and average conditions. on parallel test sets, we measure the use of both correct reference context sentences (section 6.1) and machinetranslated ones (section 8). on the contrastive test sets, only reference translations are used, as is standard practice, since target context coherence requirements prevent the use of non-reference context translations for fair evaluations (see the discussion in section 8). the prepended target-language data will need to be processed by the source language encoder under this approach, which might generate unwarranted noise. we hypothesise however that the encoder will essentially treat foreign language subwords as tokens to be copied directly into the target language, a typically simple operation for standard nmt models. we use bpe models jointly learned on merged source and target language data to facilitate this part of the process. overall, the proposed approach provides the means to exploit target language data on the decoder side, without any change to model architecture, while introducing data that might be easily processed via copying on the source side. ## using back-translated data when document-level parallel data are lacking, monolingual data in the target language can be exploited within concatenation-based approaches via back-translation (junczys-dowmunt, 2019; @xcite . some level of degradation is expected, depending on the quality of the model used to back-translate the target data, and we also expect the models to be impacted differently: the target sentence and its back-translation would be identical for all models, as would be the original target context sentences, but the nton and the src+tgt-nton models also require back-translated context, unlike the tgt-nton model. for comparison purposes we back-translated the target side of the training data for both language pairs, using a sentence-level model trained on the parallel data, and trained the main model variants strictly on the back-translated data. 4 the results are shown in tables 8 , 9 and 10, contrasting the use of parallel (pa) and back-translated (bt) data. the overall degradation using bt data was more salient in eu-es than in en-ru, which is likely due to the differences in training data size and the resulting quality of the respective models. in both cases, the x-tgt-nton variants proved more robust than the nton model. this is likely due to the latter having as context only the back-translation of the target context, while the former contain, alone or in combination with the back-translation, the original target context. overall, the tendencies observed using parallel data were replicated with back-translated data, with the src+tgt-nton model being the topperforming variant across the board, and the tgtnton a close second on target-context phenomena but performing worse than the nton model in accuracy on the gdr-src+tgt and ellipsis infl. with bt data. perhaps more surprising are the results achieved by the src+tgt-nton model, trained on bt data, on the lexical cohesion test set, where it outperformed the same variant trained on parallel data by 13 points. additional datasets might be warranted to further assess the tendencies for these models, but the results on the available datasets in terms of accuracy seem to indicate that the use of bt data is viable, and particularly exploitable by the x-tgt-nton models overall. we conjecture that this is mainly due to the fact that these approaches promote target language data which are in essence correct, while discarding or reducing the role of source context data which are likely to feature back-translation errors. ## machine-translated target context following standard practice, so far we used the reference target context instead of the machinetranslated output in our evaluations. this is meant to remove potential noise in terms of context translation errors and evaluate the approaches on their translation accuracy given a correct context. using reference translations also allows for an evaluation of phenomena where more than one context translation would be correct -e.g. box translated as boîte (fem.) instead of carton (masc.) in french -but the contrastive evaluation relies on one of these translations being selected and contextual phenomena, such as coherence, are evaluated accordingly. a correct but different context translation would unfairly affect the evaluation. still, in practice, at inference time there are no reference translations, of course. whereas x-to1 the results are shown in using mt output resulted in a slight degradation for eu-es, with results on a par with the sentencelevel baseline and at most 1.01 points loss compared to the use of reference translations. for en-ru, all models achieved comparable results except those that relied on reference translations, with gains of approximately 1 point for the latter. as previously noted, the bleu metric is known to be deficient for context-aware model evaluation, and contrastive tests provide more precise benchmarks. however, measuring mt context in terms of contrastive accuracy is not a valid option, as challenge tests rely on specific context translation choices, and the reference context is provided instead in standard practice. note that nton models would also be impacted in terms of contrastive accuracy, since mt output would affect decoding. 5 evaluating approaches based on promoting target data in a practical scenario with imperfect machine-translated context thus faces important limitations with current document-level evaluation protocols. a proper assessment of the impact of machine-translated context would also need to take into account the quality of the translation model itself, with larger models expected to minimise context translation errors in this type of approach. we leave these aspects for future research. ## accuracy at distance the results so far were measured considering context as a whole. to achieve a more fine-grained view of the differences between approaches, we computed their accuracy in terms of the distance between the current sentence and the disambiguating context information, expressed in number of sentences. the results are shown in tables 12 and 13 , indicating the distance and the percentages of cases in the corresponding dataset. the main observable tendency is that of the decreasing accuracy over distance for the nton model, in all cases but gdr-src+tgt at distance 2 (where all models perform better), in contrast with the significantly more robust accuracy of the src+tgt-nton model at larger distances, for basque-spanish in particular. the tgtnton model exhibits mixed tendencies, improving or maintaining accuracy over distance 1 in some cases, but also degrading at larger distances (gdr-src+tgt or coh-tgt, at @xmath0. note though that larger distances are under-represented in the basque-spanish test sets, and may thus not be as representative. ## conclusions in this work, we investigated the promotion of target context data within a standard concatenationbased approach to context-aware neural machine translation. the main incentive revolves around the fact that, for most contextual phenomena of interest for document-level machine translation, the relevant information is either in the target language or distributed on the source and target sides. we studied simple model variants where target context sentences are concatenated to the source sentence, either in isolation or in combination with the source context. our results in basque-spanish and english-russian, over five datasets showcasing different types of contextual phenomena, showed large improvements in terms of contrastive accuracy and bleu scores. models where the source context was discarded altogether achieved parity or slightly underperformed on phenomena involving both source and target contexts. the variants based on augmenting the source context with target data achieved the best results across the board and were also shown to be more accurate in handling context at larger distances. we further evaluated the use of back-translated data, with models merging target and source matching or outperforming variants trained on parallel data. we also measured the impact of using machine-translated context, although only in a limited way given current evaluation protocols for context-aware models, with slight degradation observed in terms of bleu. the use of more robust baseline models, trained on larger volumes of data, could mitigate the observed effects. the proposed approach promoting target data requires no changes to the standard nmt architecture and provides significant gains over strong baselines. although it also implies larger contexts when merging source and target context, it might be worth further exploring this type of approach and the respective roles of source and target context data in neural machine translation.