Neural Machine Translation Using Extracted Context Based on Deep Analysis for the J apanese- E nglish Newswire Task at WAT 2020 (2020)

Abstract: This paper describes the system of the NHK-NES team for the WAT 2020 Japanese–English newswire task. There are two main problems in Japanese-English news translation: translation of dropped subjects and compatibility between equivalent translations and English news-style outputs. We address these problems by extracting subjects from the context based on predicate-argument structures and using them as additional inputs, and constructing parallel Japanese-English news sentences equivalently translated from English news sentences. The evaluation results confirm the effectiveness of our context-utilization method.

## introduction the nhk and nhk-es team (nhk-nes) participated in the japanese-english newswire translation task at wat 2020 @xcite . currently, there are two main problems in the machine translation of news from japanese to english. one problem is that, in japanese, some subjects that can easily be guessed tend to be omitted, because of the context. in contrast, in english, subjects are needed because grammatical rules do not allow them to be omitted. these phenomena are caused by differences between the characteristics of the two languages. the other problem is the compatibility between equivalent translations and english newsstyle outputs. parallel sentences extracted from japanese news articles and english news articles are of low quality, in terms of bilingual equivalence, and contain substantial bilingual noise. this is because producing an english news article from the content of a japanese news article is not sentence-level translation. rather, it is news writing, in which the structure of the article may be changed because of differences between the structures of news in japanese and english, and the content is often summarized and supplemented because of differences in the background knowledge of the target readers. a neural machine translation (nmt) system trained on such data is supposed to output english sentences written in news style. however, the input articles tend not to be translated equivalently because omissions frequently occur unintentionally. at wat 2019, to realize equivalent translation, we constructed parallel sentence pairs-japanese news sentences that were equivalently translated to english by translators-and used these pairs for training @xcite . we confirmed that this method improved the performance of translating input sentences equivalently and that it was effective in terms of adequacy. however, the style of english produced by translators ("translated english") was different from the style of english news ("news english"). the nmt system trained on these data output english sentences in the style of translated english, and the resulting outputs had low similarity (bleu score) to the reference translations. we aim to output text in the style of news english rather than translated english. therefore, our objective is to achieve both equivalent translation and output in the style of news english. in this paper, we propose a method that analyzes the predicate-argument structures of the context of input sentences, extracts the topic in the context-based on the analyses-and uses it as the context of the input sentence to translate ( § 2). we show the difference between the translated-english and news-english styles, and construct japanese-english parallel news sentences that are equivalently translated from english news to japanese, rather than from japanese ## using extracted context based on predicate-argument structure some subjects that can easily be guessed tend to be omitted in japanese. in english, however, subjects cannot be omitted grammatically, except in the imperative form. for this reason, it is necessary to insert the implicit subjects when translating from japanese to english. in news articles, the previous context often contains information that should be added. figure 1 shows an example of a japanese news sentence with an omitted subject, and its context. implicit subjects of sentences can often be easily guessed from the context, especially from topics and subjects in the context. therefore, the preceding topics or subjects are thought to be useful as contextual information for translation. an nmt method (2-to-1) uses context by concatenating the previous source sentence to the input sentence, with the insertion of a delimiter token @xcite . however, as shown in figure 2 , a large amount of contextual information is input, and it is difficult to learn to identify and extract the necessary information by backpropagation from the target sentence. 1 there are methods of complementing subjects of input sentences for machine translation by inferring zero subjects and pre-editing the input sentences @xcite @xcite . however, these methods can also be problematic because pre-edit errors in the input sentences directly cause translation errors. subject complement is not easy and is not sufficiently accurate. to alleviate the pre-edit error of input sentences, some methods have been proposed that use pre-edited source sentences as the targets of reconstruction @xcite . however, these methods cannot solve the problem because choosing an output that reproduces incorrect pre-edited inputs degrades the output quality. furthermore, in these methods, nmt-which is used to produce nbest hypotheses-does not use context. therefore, nmt may not produce hypotheses that include appropriate contextual information. we propose a method of extracting topics and subjects, which are sometimes useful as contextual information for translation, from context by analyzing syntactic and predicate-argument structure, and adding them to the input sentences as contextual information. this avoids the difficulty-in learning to find and extract the necessary information from the context-that arises when nmt is trained by backpropagation. nmt learns whether to utilize or ignore the extracted contextual information, and how to utilize it, using source and target language information. because the proposed method does not pre-edit input sentences, it avoids the problem that the errors of subject completion are directly reflected in the translation when input sentences are pre-edited. the proposed method is described in detail below. we use the following process to extract topics or subjects from context, up to the previous sentence, and add them as contextual information to the input sentence. an example of the process is shown in figure 3 . the number of contextual sentences that can be used is not restricted. if any topic or subject is not extracted from the sentence immediately preceding the input sentence, we use a topic or subject extracted from an earlier sentence. for example, in the case that the sixth sentence of an article is an input sentence and any topic or subject is not extracted from the second to fifth sentences of the article, we use the topic or subject extracted from the first sentence. at wat 2019, we used 0.22 m pairs of japanese news sentences and equivalently translated english sentences (translated english) @xcite . the evaluation confirmed that the translated sentences were highly adequate. there were some differences in the characteristics (style) of the translated english and the english used in real news (news english). to examine the differences between the translated-english and news-english styles, we computed the test set perplexity of the two language models: one that was trained using translated-english sentences and one trained using news-english sentences. we used the cmu-cambridge statistical language modeling toolkit v2 foot_1 with the settings of 4-gram and good-turing discounting. the models were trained with 6.0 m words of translated english and 7.3 m words of news english. a test set comprising 48 k words of translated english and a test set comprising 57 k words of news english were used for the test. the results are shown in the test set perplexity (175.5) of the news-english test data was worse than that (131.7) of the translated-english test data for the model trained on the translated-english data. conversely, the test set perplexity (67.4) of the news-english test data was better than that (257.6) of the translated-english test data for the model trained on the news-english data. this inversion of these test set perplexities demonstrates that there is a difference between the translated-english style and the news-english style. to obtain translations in news-english style, we constructed equivalently translated parallel news sentences whose english style was news english. the construction method was that translators rewrote the japanese news sentences to equivalently match the corresponding english news sentences in japanese-english news articles, such articles contain substantial noise in the bilingual relationship. we constructed the parallel news sentences using jiji news (provided by jiji press). our system used 98 k sentence pairs, constructed in this manner, for training. this strategy made it possible to learn how to perform equivalent japanese-english news translation in the news-english style. in this section, we provide a brief overview of the japanese-english newswire translation task. this task started at wat 2017. participants train the translation system on the training data and translate the test data. they then submit their translations to the workshop and the workshop organizers evaluate the submitted translations. at wat 2020, a new test set, which was extracted from jiji news in japanese and english, was added. the new test set consists of test sentences, their reference sentences, and context data of the test sentences. the context data are the articles from which the test sentences were extracted. participants can use the context data to translate test sentences. the official dataset and their sizes are shown in ## results and discussion we submitted twe system outputs of the nmt system using context (nmt with context ) and the nmt system not using context (nmt without context ). official results that were evaluated by the wat 2020 organizers are shown in next, for the 200 sentences officially assessed by human evaluators, we compared the results with and without the use of context. for each out-put sentence, two adequacy scores were awarded by each of two evaluators. we define win, loss, and tie by comparing the evaluators' scores of nmt with context to those of nmt without context , as follows. win both evaluators' scores are better for nmt with context than for nmt without context , or one evaluator's score is better and the other evaluator's score is the same. loss both evaluators' scores are worse for nmt with context , or one evaluator's score is worse and the other evaluator's score is the same. tie any combination other than the above. the government aims to finalize the policy by the end of march and start utilizing the funds in autumn 2019. output (nmtwith context) the government plans to formalize the plan by the end of march, aiming to put it into operation in autumn 2019. output (nmtwithout context) the ministry plans to make an official decision by the end of march, aiming to put the system into operation in autumn 2019. input (nmtwith trump replied that he will have relevant officials discuss the matter. output (nmtwithout context) trump replied that he will have related parties discuss the matter. and outputs of nmt without context . our complements are not restricted to pronouns. the first three examples require context to translate. the added contextual information affected the outputs of nmt with context . in contrast, translation of the last input does not require context. in this case, the added contextual information seems to be ignored in nmt with context . we evaluated the subject complement. in jiji news in japanese, the subject tends to be omitted in the second sentence of each article. therefore, we examined the complements to the second sentences of each article. we used 150 articles from the context of the test data. for comparison with the 2-to-1 method, we also conducted translation using that method. the results are shown in the submitted results are in news-english style. we believe that this is not the optimal style for adequacy because there is more training data for equivalent translations in translated english than for equivalent translations in news english. we @xmath0, instead @xmath1, to evaluate the output in the translated-english style. we translated these test sentences using the nmt with context model. nmt with context adequacy represents the outputs in the translated english style. the results are shown in tables 9 . we submitted nmt with context adequacy to the wat automatic evaluation server and published it. the jpo adequacy in ## conclusion we described our method using context, the resources used for training, and the system setup for the japanese-english newswire translation task at wat 2020. we proposed the extraction of topics and subjects from context, based on deep analysis, and their use as the context of input sentences. we also constructed a set of parallel japanese-english news sentence pairs by equivalently translating english news sentences to japanese sentences. this resource enabled us to directly learn equivalent translation with the news-english style. the evaluation results confirmed the effectiveness of our method.