Effective Use of Target-side Context for Neural Machine Translation (2020)

Abstract: In this paper, we deal with two problems in Japanese-English machine translation of news articles. The first problem is the quality of parallel corpora. Neural machine translation (NMT) systems suffer degraded performance when trained with noisy data. Because there is no clean Japanese-English parallel data for news articles, we build a novel parallel news corpus consisting of Japanese news articles translated into English in a content-equivalent manner. This is the first content-equivalent Japanese-English news corpus translated specifically for training NMT systems. The second problem involves the domain-adaptation technique. NMT systems suffer degraded performance when trained with mixed data having different features, such as noisy data and clean data. Though the existing methods try to overcome this problem by using tags for distinguishing the differences between corpora, it is not sufficient. We thus extend a domain-adaptation method using multi-tags to train an NMT model effectively with the clean corpus and existing parallel news corpora with some types of noise. Experimental results show that our corpus increases the translation quality, and that our domain-adaptation method is more effective for learning with the multiple types of corpora than existing domain-adaptation methods are.

## introduction neural machine translation (nmt) @xcite @xcite has made great progress in translating sentences in isolation. in contrast to isolated sentences, sentences in documents cannot be correctly translated without context outside the current sentence @xcite . to address this problem, various context-aware nmt models have been developed to exploit preceding and/or succeeding sentences in source-and/or target-side languages as context @xcite @xcite @xcite @xcite @xcite . @xcite used nmt models with source-and target-side contexts in the previous sentence, and reported that the source-side context was effective for improving the translation quality. however, the target-side context led to lower quality even though they highlighted the importance of the target-side context. @xcite also pointed out that the use of the target-side context increases the risk of error propagation. we considered that the reason for the low quality was a gap between a training and an inference when using target-side context in the model. at the training phase, the ground truth sentences (references) were used as the target-side context, and at the inference phase, the sentences predicted by a translation model were used. the predicted sentence includes translation errors such as mistranslation, under-translation, and over-translation not included in the ground truth sentence. even if there are no translation errors in the predicted sentence, there is a bias of translationese that is outputs of machine translation tend to be simpler and more standardized than human-translated texts @xcite . therefore, the predicted target-side sentence tends to have lower diversity than the ground truth sentence. this different use, called exposure bias @xcite , led to lower translation quality. the following example, extracted from the iwslt2017 japanese-english dataset @xcite , presents the previous target-side ground truth and predicted sentences (ground truth context and predicted context) in addition to the source sentence and target sentences (reference and vanilla nmt output). this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . ground truth context: she lays eggs, she feeds the larvae -so an ant starts as an egg, then it's a larva. predicted context: and when they get their eggs, they get their eggs, and the queen is there. source sentence: 脂肪を吐き出して幼虫を育てます reference: she feeds the larvae by regurgitating from her fat reserves. vanilla nmt output: they take fat and they raise the larvae of their larvae. this example indicates two things: (a) the context information, which is a previous target-side sentence in this example, is effective to translate the source sentence correctly, and (b) the ground truth context and predicted context are different. first, we can find that both the ground truth context and the predicted context include significant information (that is "she" for the ground truth context and "queen" for the predicted context) for a correct translation, where the subject of the source is "she." the vanilla nmt trained without the contexts fails to translate the subject of the sentence shown in the vanilla nmt output of the above example. second, the predicted context includes some translation errors of mistranslation, under-translation, and over-translation not included in the ground truth context. this means that there is the exposure bias. in this paper, we propose a controlled sampling with both the ground truth sentence and the predicted sentence of the target-side context to alleviate the gap between the training and the inference, inspired by the scheduled-sampling method @xcite . at the first epoch, we train the translation model with only the ground truth contexts, without the predicted contexts. then we change the training data for each epoch in order to gradually force the translation model to deal with noise. the predicted contexts are generated by the vanilla nmt model. we implement the proposed method on a concatenationbased context-aware nmt and a multi-encoder context-aware nmt. experimental results on english ↔ japanese translation tasks in the news corpus and iwslt2017 dataset @xcite and english ↔ german translation tasks in the iwslt2017 dataset show that our proposed method can improve on the previous model in terms of bilingual evaluation understudy (bleu) scores. ## context-aware nmt in this section, we introduce two existing techniques for context-aware nmt that have been actively developed in recent work. ## proposed method we propose a sampling method that can be applied to the context-aware nmts in the previous section to alleviate the gap between the training and inference phases in context-aware translation tasks with target-side context. ## experiments to evaluate our method, we experimented on two document-level parallel datasets in two language pairs for machine translation. ## conclusion in this work, we have presented a controlled sampling method for a context-aware neural machine translation (nmt) using target-side previous ground-truth and predicted sentences as context. our aim is to tackle the exposure bias of the target-side context between the training and the inference on the basis of the scheduled-sampling method. we implemented and evaluated the proposed method on two types of context-aware nmt systems: concatenation-based context-aware and a multi-encoder context-aware nmts. the experimental results verified the effectiveness of our proposed method in terms of bleu.