Document-level Neural MT : A Systematic Comparison (2020)

Abstract: In this paper we provide a systematic comparison of existing and new document-level neural machine translation solutions. As part of this comparison, we introduce and evaluate a document-level variant of the recently proposed Star Transformer architecture. In addition to using the traditional metric BLEU, we report the accuracy of the models in handling anaphoric pronoun translation as well as coherence and cohesion using contrastive test sets. Finally, we report the results of human evaluation in terms of Multidimensional Quality Metrics (MQM) and analyse the correlation of the results obtained by the automatic metrics with human judgments.

## introduction there has been undeniable progress in machine translation (mt) in recent years, so much so that for certain languages and domains, when sentences are evaluated in isolation, it has been suggested that mt is on par with human translation @xcite . however, it has been shown that human translation clearly outperforms mt at the document level, when the whole translation is taken into account @xcite @xcite . for example, the conference on machine translation (wmt) now considers inter-sentential translations in their shared task @xcite . this sets a demand for context-aware machine translation: systems that take the context into account when translating, as opposed to translating sentences independently. translating sentences in context (i.e. at the document level) is essential for correctly handling discourse phenomena whose scope can go beyond the current sentence and which therefore require document context @xcite @xcite . important examples include anaphora, lexical coherence and cohesion, deixis and ellipsis; crucial aspects in delivering high quality translations which often are poorly evaluated using standard automatic metrics. numerous context-aware neural mt (nmt) approaches have been proposed in recent years @xcite @xcite @xcite , integrating source-side and sometimes target-side context. however, they have often been evaluated on different languages, datasets, and model sizes. certain models have also previously been trained on few sentence pairs rather than in more realistic, highresource scenarios. a direct comparison and analysis of the methods, particularly concerning their individual strengths and weaknesses on different language pairs is therefore currently lacking. we fill these gaps by comparing a representative set of context-aware nmt solutions under the same experimental settings, providing: ## context-aware nmt methods we compare three previous context-aware approaches (concatenation, multi-source and cachebased) in our experiments. as well as illustrating different methods of integrating context, they vary in terms of which context (source/target, previous/future) and how much context (number of sentences) they can exploit, as shown in concatenation: tiedemann and scherrer (2017) use the previous sentence as context, i.e. x (k-1) and y (k-1) , concatenated to the current sentence, i.e. x (k) and y (k) , separated by a special token. it is called 2to1 when just the source-side context is used, and 2to2 when the target is used too. ## doc-star-transformer we propose a scalable approach to document-level nmt inspired by the star architecture @xcite for sentence-level nmt. we have an equivalent relay node and build sentence-level representations; we propagate this non-local information at document-level and enrich the word-level embeddings with context information. to do this, we augment the vanilla sentencelevel transformer model of @xcite with two additional multi-headed attention sublayers. the first sub-layer is used to summarize the global contribution of each sentence into a single embedding. the second layer then uses these sentence embeddings to update word representations throughout the document, thereby incorporating document-wide context. in §4.1, we describe our model assuming it can attend to context from the entire document without practical memory constraints. then in §4.2 we show how to extend the model to arbitrarily long contexts by introducing sentence-level recurrence. ## evaluating context-aware nmt the evaluation of context-aware mt is notoriously tricky @xcite ; standard automatic metrics such as bleu @xcite are poorly suited to evaluating discourse phenomena (e.g. anaphoric references, lexical cohesion, deixis, ellipsis) that require document context. we therefore evaluate all models using a range of phenomenon-specific contrastive test sets. contrastive sets are an automatic way of evaluating the handling of particular phenomena @xcite . the aim is to assess how well models rank correct translations higher than incorrect (contrastive) ones. for context-aware test sets, the correctness of translations depends on context. several such sets exist for a range of discourse phenomena and for several language directions: @xmath0, @xmath1. in this article, we evaluate using the following test sets for our two language directions of focus, @xmath2: en-fr: anaphora, lexical choice @xcite ): 3 two manually crafted sets (200 contrastive pairs each), for which the previous sentence determines the correct translation. the sets are balanced such that each correct translation also appears as an incorrect one (a non-contextual baseline achieves 50% precision). anaphora examples include singular and plural personal and possessive pronouns. in addition to standard contrastive examples, this set also contains contextually correct examples, where the antecedent is translated strangely, designed to test the use of past translation decisions. lexical choice examples include cases of lexical ambiguity (cohesion) and lexical repetition (cohesion). @xmath3: anaphoric pronouns (contrapro) (m üller et al., 2018). 4 a large-scale automatically created set from opensubtitles2018 @xcite , in which sentences containing the english anaphoric pronoun it (and its corresponding german translations er, sie or es) are automatically identified, and contrastive erroneous translations are automatically created. the test set contains 4,000 examples for each target pronoun type, and the disambiguating context can be found in any number of previous sentences. ## experimental setup as mentioned in §1, we aim to provide a systematic comparison of the approaches over the same datasets, training data sizes and language pairs. we study whether pre-training with larger resources (in a more realistic high-resource scenario) has an impact on the methods on language directions that are challenging for sentence-level mt. we consider translation from english into french (fr), german (de) and brazilian portuguese (pt br), which all have gendered pronouns corresponding to neutral anaphoric pronouns in english (it for all three and they for fr and pt br). we compare the three previous methods ( §3) plus the doc-star-transformer in two scenarios: (i) document mt, testing on ted talks (@xmath4, and (ii) chat mt testing on proprietary conversation data for all three directions. ## conclusion we provided a systematic comparison of several context-aware nmt methods. one of the methods in this comparison was a new adaptation of the recently proposed startransformer architecture to document-level mt. in addition to bleu, we reported results of the contrastive evaluation of context-dependent phenomena (anaphora and coherence/cohesion), creating an additional largescale contrastive test set for @xmath5, and we carried out human evaluation in terms of multidimensional quality metrics (mqm). our findings suggest that existing context-aware approaches are less advantageous in scenarios with larger datasets and strong sentencelevel baselines. in terms of the targeted contextdependent evaluation, one of the promising ap-proaches is one of the simplest: the concat2to2, where translated context is channelled through the decoder, although our doc-star-transformer method achieves good results according to the manual evaluation of mt quality.