Contrastive Learning for Context-aware Neural Machine Translation Using Coreference Information (2021)

Abstract: Context-aware neural machine translation (NMT) incorporates contextual information of surrounding texts, that can improve the translation quality of document-level machine translation. Many existing works on context-aware NMT have focused on developing new model architectures for incorporating additional contexts and have shown some promising results. However, most of existing works rely on cross-entropy loss, resulting in limited use of contextual information. In this paper, we propose CorefCL, a novel data augmentation and contrastive learning scheme based on coreference between the source and contextual sentences. By corrupting automatically detected coreference mentions in the contextual sentence, CorefCL can train the model to be sensitive to coreference inconsistency. We experimented with our method on common context-aware NMT models and two document-level translation tasks. In the experiments, our method consistently improved BLEU of compared models on English-German and English-Korean tasks. We also show that our method significantly improves coreference resolution in the English-German contrastive test suite.

## introduction neural machine translation (nmt) has achieved impressive performances on translation quality, due to the introduction of novel deep neural network (dnn) architectures such as encoder-decoder model @xcite , and self-attentional networks like transformer @xcite . the state-of-the-art nmt systems are now even comparable with human translators in sentence-level performance. however, there are a number of issues on document-level translation @xcite . these include pronoun resolution across sentences @xcite , which needs cross-sentential contexts. to incorporate such document-level con-textual information, several methods for contextaware nmt have been recently proposed. many of the works have focused on introducing new model architectures like multi-encoder models @xcite for encompassing contextual texts of the source language. these works have shown significant improvement in addressing discourse phenomena such as anaphora resolution mentioned above, as well as moderate improvements in overall translation quality @xcite . despite some promising results, most of the existing works have trained the model by minimizing cross-entropy loss, making the model rather exploit contextual information implicitly such as a form of regularization @xcite . data augmentation for context-aware nmt is also an important issue, despite that recent works have focused on back-translation @xcite . in this paper, we propose a coreference-based contrastive learning for context-aware nmt (corefcl), a novel data augmentation and contrastive learning scheme leveraging coreference information. cross-sentential coreference between the source and target sentence can be a good source of training signal for context-aware nmt since it occurs when one or more expressions refer to the same entity, thus reflects dependencies between the source and contextual sentences. corefcl starts by conducting automatic annotation of coreference between the source and contextual sentences. then, the referred mentions on contextual sentences are corrupted by removing and/or replacing tokens to generate contrastive examples. with those contrastive examples, we introduce a contrastive learning scheme equipped with a max-margin loss which encourages the model to discriminate between the original examples and the contrastive ones. by doing so, corefcl makes the model more sensitive to cross-sentential contextual information. we experimented with corefcl on three english-german corpora and one english-korean document-level corpus, including wmt, iwslt ted talk, and opensubtitles'18 english-german subtitles translation task, and a web-crawled english-korean subtitles translation. in all translation tasks, corefcl consistently improves overall bleu over baseline models without corefcl. on experiments with three common context-aware model settings, we show that improvements by corefcl are also model-agnostic. finally, we show that the proposed method significantly improved the performance on contrapro @xcite , an english-german contrastive coreference benchmark. in particular, @xcite introduced a context-aware transformer model which is able to induce anaphora relations, @xcite showed that a model using cross-sentential contextual information significantly outperforms in document-level translation tasks, and @xcite insisted that context-aware models record the best performance especially in spoken language translation tasks where mandatory information tend to be sparse over multiple sentences. the simplest method for context-aware machine translation is to concatenate all surrounding sentences and treat the concatenated sequence as a single sentence @xcite . although the concatenation strategy boosted transformer architectures in multiple tasks @xcite @xcite , it lagged behind efficiency as the transformer architecture has limited long-range dependency @xcite . to improve the efficiency, an additional encoder module is introduced to encode only the context sentences @xcite @xcite . additionally, hierarchical structures also have been introduced because the context sentences do not have the same significance as the input sentences @xcite . ## context-aware nmt models in this section, we briefly overview context-aware nmt methods and describe our baseline models which are also commonly adopted in recent works. generally, a sentence-level (context-agnostic) nmt model takes an input sentence in a source language and returns an output sentence in a target language. on the other hand, a context-aware nmt model is designed to handle surrounding contextual sentences of source and/or target sentences. we focus on leveraging the contextual sentences of the source language. throughout this work, we consider the transformer @xcite as a base model architecture by following the majority of the recent works on context-aware nmt. transformer consists of a stack of self-attentional layers in which a self-attention module is followed by a feedforward module for each layer. here we list four transformer-based configurations that we used in the experiments: all the model structures are described in figure 1 . 4 our method: corefcl in this section, we explain the main idea of core-fcl, a data augmentation and contrastive learning scheme leveraging coreference between the source and contextual sentences. ## conclusions and future work we have presented a data augmentation and contrastive learning scheme based on coreference for context-aware nmt. by leveraging coreference mentions between the source and target sentence, corefcl effectively generates contrastive examples for applying contrastive learning on contextaware nmt models. in the experiments, corefcl consistently improves the translation quality and pronoun resolution accuracy. as future work, we plan to extend corefcl to target contexts since maintaining coreference consistency needs both the source and the target contexts. it would be also interesting that applying corefcl for fine-tuning pre-trained big language models like bart @xcite or t5 @xcite for downstream document-level mt tasks.