Contextual Neural Machine Translation Improves Translation of Cataphoric Pronouns (2020)

Abstract: The advent of context-aware NMT has resulted in promising improvements in the overall translation quality and specifically in the translation of discourse phenomena such as pronouns. Previous works have mainly focused on the use of past sentences as context with a focus on anaphora translation. In this work, we investigate the effect of future sentences as context by comparing the performance of a contextual NMT model trained with the future context to the one trained with the past context. Our experiments and evaluation, using generic and pronoun-focused automatic metrics, show that the use of future context not only achieves significant improvements over the context-agnostic Transformer, but also demonstrates comparable and in some cases improved performance over its counterpart trained on past context. We also perform an evaluation on a targeted cataphora test suite and report significant gains over the context-agnostic Transformer in terms of BLEU.

## introduction standard machine translation (mt) systems typically translate sentences in isolation, ignoring essential contextual information, where a word in a sentence may reference other ideas or expressions within a piece of text. this locality assumption hinders the accurate translation of referential pronouns, which rely on surrounding contextual information to resolve cross-sentence references. the issue is further exacerbated by differences in pronoun rules between source and target languages, often resulting in morphological disagreement in the quantity and gender of the subject being referred to @xcite . rapid improvements in nmt have led to it replacing smt as the dominant paradigm. with this, context-dependent nmt has gained traction, overcoming the locality assumption in smt through the use of additional contextual information. this has led to improvements in not only the overall translation quality but also pronoun translation @xcite @xcite . however, all these works have neglected the context from future sentences, with @xcite reporting it to have a negative effect on the overall translation quality. in this work, we investigate the effect of future context in improving nmt performance. we particularly focus on pronouns and analyse corpora from different domains to discern if the future context could actually aid in their resolution. we find that for the subtitles domain roughly 16% of the pronouns are cataphoric. this finding motivates us to investigate the performance of a context-dependent nmt model @xcite trained on the future context in comparison to its counterpart trained on the past context. we evaluate our models in terms of overall translation quality (bleu) and also employ three types of automatic pronountargeted evaluation metrics. we demonstrate strong improvements for all metrics, with the model using future context showing comparable or in some cases even better performance than the one using only past context. we also extract a targeted cataphora test set and report significant gains on it with the future context model over the baseline. domain #sentences document length english-german subtitles 9.39m/9k/14.1k 565.8/582.2/591.0 europarl 1.67m/3.6k/5.1k 14.1/15.0/14.1 ted talks 0.21m/9k/2.3k 120.9/96.4/98.7 english-portuguese subtitles 15.2m/16.1k/23.6k 580.4/620.6/605.3 context-aware nmt multiple works have successfully demonstrated the advantages of using larger context in nmt, where the context comprises few previous source sentences @xcite , few previous source and target sentences @xcite , or both past and future source and target sentences @xcite @xcite . further, context-aware nmt has demonstrated improvements in pronoun translation using past context, through concatenating source sentences @xcite or through an additional context encoder @xcite @xcite . @xcite observed reasonable improvements in generic and pronoun-focused translation using three previous sentences as context. @xcite observed improvements using the previous sentence as context, but report decreased bleu when using the following sentence. we, on the other hand, observe significant gains in bleu when using the following sentence as context on the same data domain. ## contextual analysis of corpora to motivate our use of the future context for improving the translation of cataphoric pronouns in particular and nmt in general, we first analyse the distribution of coreferences for anaphoric and cataphoric pronouns over three different corpora -opensubtitles2018 foot_0 @xcite , europarl @xcite and ted talks @xcite -for english-german. for europarl and ted talks, we use the publicly available document-aligned version of the corpora @xcite . for subtitles, we align the english and german subtitles at the document-level using publicly available alignment links. 2 to control for the length and coherency of documents, we keep subtitles with a run-time less than 50 minutes (for english) and those with number of sentences in the hundreds. the corpus is then randomly split into training, development and test sets in the ratio 100:1:1.5. ## experiments datasets we experiment with the subtitles corpus on english-german and english-portuguese language-pairs. to obtain english-portuguese data, we employ the same pre-processing steps as reported in §3 (corpus statistics are in the moses toolkit @xcite and split into subword units using a joint bpe model with 30k merge operations @xcite . 5 description of the nmt systems as our baseline, we use the dynet @xcite implementation of transformer @xcite . 6 for the context-dependent nmt model, we choose the transformer-han encoder @xcite , which has demonstrated reasonable performance for anaphoric pronoun translation on subtitles. we extend its dynet implementation @xcite to a single context sentence. 78 for training, transformer-han is initialised with the baseline transformer and then the parameters of the whole network are optimised in a second stage as in @xcite (details of model configuration are in appendix a.1). for evaluation, we compute bleu @xcite on tokenised truecased text and measure statistical significance with @xmath0.005 @xcite . ## conclusions in this paper, we have investigated the use of future context for nmt and particularly for pronoun translation. while previous works have focused on the 13 it should be noted that the cataphora test set is extracted based on the existence of cataphoric-pairs in the english-side, which may have biased the evaluation when english was in the target. 14 attention is average of the per-head attention weights. eva vanmassenhove, christian hardmeier, and andy way. 2018. getting gender right in neural machine translation. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 3003-3008, brussels, belgium. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. elena voita, pavel serdyukov, rico sennrich, and ivan titov. 2018. context-aware neural machine translation learns anaphora resolution. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1264-1274, melbourne, australia. association for computational linguistics. longyue wang, zhaopeng tu, andy way, and qun liu. 2017. exploiting cross-sentence context for neural machine translation. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2826-2831, copenhagen, denmark. association for computational linguistics. jiacheng zhang, huanbo luan, maosong sun, feifei zhai, jingfang xu, min zhang, and yang liu. 2018. improving the transformer translation model with document-level context. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 533-542, brussels, belgium. association for computational linguistics.