Context-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model (2021)

Abstract: Although many end-to-end context-aware neural machine translation models have been proposed to incorporate inter-sentential contexts in translation, these models can be trained only in domains where parallel documents with sentential alignments exist. We therefore present a simple method to perform context-aware decoding with any pre-trained sentence-level translation model by using a document-level language model. Our context-aware decoder is built upon sentence-level parallel data and target-side document-level monolingual data. From a theoretical viewpoint, our core contribution is the novel representation of contextual information using point-wise mutual information between context and the current sentence. We demonstrate the effectiveness of our method on English to Russian translation, by evaluating with BLEU and contrastive tests for context-aware translation.

## introduction neural machine translation (nmt) has typically been explored in sentence-level translation settings. such sentence-level nmt models inevitably suffer from ambiguities when a source sentence has multiple plausible interpretations. examples of such ambiguities include anaphora, ellipsis, and lexical coherence @xcite ; although resolving these ambiguities has only a minor impact on the translation performance measured by bleu scores @xcite , they are vital in smoothly reading the translated documents. to address this issue, context-aware nmt models which incorporate document-level information in translation have recently been explored (jean et al., 2017; wang et al., 2017; tiedemann and scherrer, 2017; maruf and haffari, 2018; voita et al., 2018; bawden et al., 2018; miculicich et al., 2018; maruf et al., 2019; voita et al., 2019b; yu et al., 2020; * currently at mitsubishi ufj morgan stanley securities @xcite . most of these models are end-to-end models that require document-level parallel data with sentential alignments for training. however, this data is available in only a few domains @xcite . researchers have therefore started to utilize target-side monolingual data to construct auxiliary models which help a sentence-level nmt model perform context-aware translation @xcite @xcite . in this study, we propose a simple yet effective approach to context-aware nmt using two primitive components, a sentence-level nmt model and a document-level language model (lm). we can independently train the two components on common sentence-level parallel data and documentlevel monolingual data, respectively, without using document-level parallel data. our approach thereby makes it possible to perform context-aware translation with any pre-trained sentence-level nmt model, using a pre-trained document-level lm. to give a probabilistic foundation to this combination of two independent models, we exploit the probabilistic nature of nmt decoding. when generating a sequence, a left-to-right decoder outputs a categorical probability distribution over the vocabulary at every time step. the decoder assigns higher probabilities to the tokens that would be more suitable at that step. therefore, when multiple valid translations are possible for the source sentence, the decoder just gives a higher probability to the translation that is plausible without considering contexts. we thus adjust the probability distributions in a context-aware manner using a target-side document-level lm which models inter-sentential dependencies in the target-side document. we evaluate our methods on english to russian translations with the opensubtitles2018 corpus @xcite in terms of the bleu scores and contrastive discourse test sets @xcite . experimental results confirm that our method achieved comparable performance with existing context-aware nmt models that require either document-level parallel data @xcite or more than one additional model @xcite for capturing contexts in translation. the contributions of this paper are as follows: in this section, assuming a sentence-level encoderdecoder model @xcite , we first derive context-aware score (c-score for short), a context-aware objective function of outputs to be maximized in decoding. we then describe how to compute the c-score using the decoder with a document-level language model (d-lm) ( § 2.1). we finally detail how to perform context-aware decoding based on c-score ( § 2.2). ## experimental setup we evaluate our methods on english to russian translation, in terms of bleu scores @xcite and contrastive tests @xcite . ## related work the effectiveness of incorporating context into translation was shown in earlier literature on document-level nmt @xcite using the single encoder architecture. multi-encoder architectures were explored to better capture contextual information @xcite @xcite @xcite @xcite @xcite . however, since parallel data is often constructed by picking up reliable sentential alignments from comparable documents, documentlevel sentence-aligned parallel data for training these document-level nmt models are expensive to obtain and available in only a few domains and language pairs @xcite . recent studies have therefore started to focus on modeling contexts using document-level monolingual data. the current approaches are grouped into three categories: data augmentation via back-translation @xcite , a post-editing model @xcite , and modeling document-level fluency via document-level lms @xcite @xcite . in what follows, we review these approaches in detail. @xcite reported that the data augmentation by back-translation @xcite enhances a document-level nmt model with a single encoder architecture in lowresource settings. however, we have obtained limited improvements in our settings ( finally, we noticed that jean and cho (2020) (which appeared after the preprint version of this paper (sugiyama and yoshinaga, 2020) foot_7 had been submitted) have reached a formulation that is very similar to the one presented in this paper by reformulating a noisy channel model of bayes docreranker @xcite . concrete differences between our work and theirs include the fact that we conducted thorough analysis on the performance of different decoding strategies (not only beam search but also reranking). we also interpreted the subtraction of lm scores as point-wise mutual information and analyzed it by observing pmi correlation between source and target pmi to deepen the understanding of the formulation. ## conclusions we present an approach to context-aware nmt based on pmi between the context and the current sentence. we first provide the formulation of the objective, c-score, and the computation process of the c-score using a sentence-level translation model and a document-level language model. we investigate two search methods, reranking and beam search, and evaluate the methods for english-russian translation. we also provide some analysis and visualization to better understand the nature of pmi between the context and the current sentence. we plan to design context-aware bleu using pmi for evaluating context-aware nmt models. we will evaluate our method on non-autoregressive nmt @xcite . we will release all code and data to promote the reproducibility of results. 9