Diving Deep into Context-Aware Neural Machine Translation (2020)

Abstract: Context-aware neural machine translation (NMT) is a promising direction to improve the translation quality by making use of the additional context, e.g., document-level translation, or having meta-information. Although there exist various architectures and analyses, the effectiveness of different context-aware NMT models is not well explored yet. This paper analyzes the performance of document-level NMT models on four diverse domains with a varied amount of parallel document-level bilingual data. We conduct a comprehensive set of experiments to investigate the impact of document-level NMT. We find that there is no single best approach to document-level NMT, but rather that different architectures come out on top on different tasks. Looking at task-specific problems, such as pronoun resolution or headline translation, we find improvements in the context-aware systems, even in cases where the corpus-level metrics like BLEU show no significant improvement. We also show that document-level back-translation significantly helps to compensate for the lack of document-level bi-texts.

## introduction even though machine translation (mt) has greatly improved with the emergence of neural machine translation (nmt) @xcite and more recently the transformer architecture @xcite , there remain challenges which can not be solved by using sentence-level nmt systems. among other issues, this includes the problem of inter-sentential anaphora resolution @xcite or the consistent translation across a document @xcite , for which the system inevitably needs document-level context information. in recent years, many works have focused on changing existing nmt architectures to incorpo-rate context information in the translation process @xcite @xcite . however, often times results are reported only on very specific tasks (most commonly subtitle translation), making it difficult to assess the potential of the different methods in a more general setting. this, together with the fact that big improvements are typically reported on low resource tasks, gives the impression that documentlevel nmt mostly improves due to regularization rather than from leveraging the additional context information. in this work we want to give a more complete overview of the current state of documentlevel nmt by comparing various approaches on a variety of different tasks including an applicationoriented e-commerce setting. we discuss both, widely used performance metrics, as well as highly task-specific observations. another important aspect when talking about document-level nmt is the applicability in "real life" settings. there, when faced with a low resource data scenario, back-translation is an established way of greatly improving system performance @xcite . however, to the best of our knowledge, the effect of backtranslation data obtained and used by context-aware models has never been explored before. the main contributions of this paper are summarized below: the discourse-or document-level translation is a long-standing and unsolved topic in the machine translation community @xcite @xcite . although neural machine translation @xcite @xcite has recently become the dominant translation paradigm that provides superior performance, the independence between sentences is still the fundamental assumption taken for granted by most nmt systems. this means, that discourse-level phenomena between sentences such as pronominal reference, consistent lexical choice, and verbal tenses, etc. can not be addressed by these sentence-level nmt systems @xcite . the current nmt approaches tackling inter-sentential discourse phenomena can be roughly categorized into three aspects, augmenting nmt by to include the source-side context, tiedemann and scherrer (2017) concatenate consecutive sentences as input to the nmt system, while @xcite ; @xcite ; @xcite use an additional encoder to extract contextual information from a few previous source-side sentences. these works only consider a local context, including a few previous sentences. some researches seek to capture the global document context; @xcite summarize the global context from all previous sentences in a document with a pre-trained hierarchical rnn and then use it for updating decoder states. very recently, @xcite proposed a discourse structure-based encoder that takes account of the discourse structure information of the input document. for adding additional target-side context, tiedemann and scherrer (2017); agrawal et al. (2018) conduct multi-sentences decoding and observe only a minor improvement. @xcite apply cache-based models to store vector representations for both source-and target-side context. similarly, @xcite augment their nmt system with an external cache to memorize the translation history. @xcite integrate two hierarchical attention networks (han) @xcite in the nmt model to take account for source and target context. @xcite apply a hierarchical attention module on sentences and words in the context to select contextual information that is more relevant to the current sentence. for incorporating document-level monolingual data from the source language, @xcite use bert @xcite to model the sourceside context and integrate it with the encoder and decoder of the nmt model. junczys-dowmunt (2019) share the parameters of a bert-style encoder trained on monolingual documents with the mt model. to utilize the document-level monolingual data from the target language, junczys-dowmunt (2019) also submit a system that trained on the combination of real and synthetic document-parallel data obtained by back-translation. however, they do not consider document-level back-translation. @xcite proposed a document-level postediting system which is trained only using the monolingual document-level corpus. recently, there has been a tendency in the community to conclude that the context used in a context-aware mt model works as regularisation or noise generator. @xcite compare several multi-encoders methods and claim that including this additional information can improve translation performance, but it is mostly due to the regularization effect rather than the contextual information. @xcite also compare some contextaware architectures by replacing the real context with some random signal and show that random signals can achieve the same level improvement as the real context. however, it should be taken with a grain of salt since solving this task, along with the analysis, is quite challenging. there are many impact factors from the architecture, the data at hand, to the metric being used for evaluation. one issue that can not be ignored in all discourserelated researches is the problem of evaluation. since some discourse-level phenomena between sentences appear less frequently, although relevant, there is doubt if the metrics like bleu score @xcite can capture these complex re-lationships @xcite . to get more insights into the capacities dealing with discourse-level phenomena of their mt models, some researchers use more targeted evaluation scores @xcite , like the accuracy of pronoun translation (apt) werlen and popescu-belis (2017), or they evaluate their systems on some specific test suites that contain more and more complex discourse phenomena @xcite @xcite . ## document-level nmt in this section, we first describe several commonly used context-aware nmt architectures and highlight the differences among them, largely following the work by @xcite . afterwards, we describe one radical attempt to represent the document-level context in one single embedding vector using bert @xcite . finally, we explain our proposed paradigm to use documentlevel back-translation in detail. note that in this work, we consistently use transformer @xcite as our basic architecture and modify it accordingly. ## conclusion in this work, we give a comprehensive comparison of current approaches to document-level nmt. to draw meaningful conclusions, we report results for standard nmt metrics on four diverse tasks -differing in the domain and the data size. we find that there is no single best approach to documentlevel nmt, but rather that different architectures work the best on various tasks. looking at taskspecific problems, such as pronoun resolution or headline translation, we find improvements in the context-aware systems, which is not visible in the corpus-level metric scores. we also investigate methods to include document-level monolingual data on both source (using pre-trained embeddings) and target (using back-translation) sides. we argue that the performance improvements from the pre-trained encoder predominantly come from increased training data and other task-specific phenomena unrelated to actual context information utilization. regarding back-translation, we find that document-level systems seem to benefit more from synthetically generated data than their sentence-level counterparts. we discuss that this is because document-level systems are more robust to sentence-level noise. we plan to expand our experiments to incorporate document-level monolingual data on both source and target sides. this makes sense just by looking at the data conditions of almost every task: document-level parallel data is scarce, but document-level monolingual data is abundant.