Studying The Impact Of Document-level Context On Simultaneous Neural Machine Translation (2021)

Abstract: In a real-time simultaneous translation setting and neural machine translation (NMT) models start generating target language tokens from incomplete source language sentences and making them harder to translate and leading to poor translation quality. Previous research has shown that document-level NMT and comprising of sentence and context encoders and a decoder and leverages context from neighboring sentences and helps improve translation quality. In simultaneous translation settings and the context from previous sentences should be even more critical. To this end and in this paper and we propose wait-k simultaneous document-level NMT where we keep the context encoder as it is and replace the source sentence encoder and target language decoder with their wait-k equivalents. We experiment with low and high resource settings using the ALT and OpenSubtitles2018 corpora and where we observe minor improvements in translation quality. We then perform an analysis of the translations obtained using our models by focusing on sentences that should benefit from the context where we found out that the model does and in fact and benefit from context but is unable to effectively leverage it and especially in a low-resource setting. This shows that there is a need for further innovation in the way useful context is identified and leveraged.

## introduction neural machine translation (nmt) @xcite is an end-to-end approach known to give the state of the art results for a variety of language pairs. in standard nmt, the entire source language sentence is fed to the model, and once the entire target language sentence is generated, it is presented to the user. however, in a real-time translation setting, translation models are expected to present translated words or phrases as they are generated. furthermore, waiting for the entire source language sentence adds to the latency, and therefore an optimal solution is to have a model that can start generating target language words right after the first few source language words are available for translation. this is known as simultaneous nmt (snmt) and is known for its poor translation quality, especially in lowresource settings. the concept of waiting for k words or tokens before generating target language words or tokens is known as wait-k snmt @xcite . in this paper, we work with the transformer architecture as the standard nmt model, consisting of a bidirectional encoder and unidirectional decoder. the decoder is able to attend to all source language tokens when generating target language tokens. however, in the case of the wait-k snmt model, the standard encoder and decoder are replaced with their snmt equivalents, which are a unidirectional encoder and a modified decoder, respectively. the decoder can only look at i + k -1 encoder tokens when predicting the i th token. we are aware of a previous work that has shown that using an image as an additional modality can help improve translation quality in a wait-k setting when k is a small value around 1 to 4 @xcite . the additional image modality provides the model with a form of context which helps disambiguate hard-to-translate phenomena, especially when needed information is not available yet during translation. an additional image modality may not always be available, and thus, taking advantage of the context in the form of previously seen sentences is the only viable option. research in document-level nmt has already proven that context from neighbouring sentences can help enhance representations and thereby improve translation quality @xcite @xcite . the simplest documentlevel nmt architecture involves using an additional encoder that encodes the context sentences, following which the encoded context is used to augment the representation of the sentence to be translated @xcite . just like using an image as a modality helps enrich the encoding of the sentence with additional disambiguation information, the context sentences might also contain such useful information. we already know that in an snmt setting, due to partial sentences being translated, the amount of context available to the decoder is limited, and thus leveraging the context sentences should significantly boost snmt translation quality. this motivated us to combine document-level nmt with snmt leading to document-level snmt. our document-level snmt architecture is simple, where we have a sentence encoder, context encoder, and a decoder except that the sentence encoder and decoder are wait-k snmt equivalents of the standard encoder and decoder. we experiment with a high-resource opensub-titles2018 dataset for @xmath0. our observations show that document-level context helps improve translation slightly in both settings but not by a large margin. we then perform a statistical and manual analysis of the translations where we observe that while snmt models definitely benefit from context, they are unable to utilize context effectively and sometimes suffer due to the provided context. this opens up the possibility of research into better mechanisms for leveraging context more effectively. ## related work for simultaneous translation, it is crucial to predict the words that have not appeared yet. mainly, snmt can mostly be implemented with fixed or adaptive policies @xcite . adaptive policy decides whether to read another source word or write a target word in one model @xcite @xcite . most dynamic models with adaptive policies @xcite @xcite focus on mechanisms that determine the optimal number of source language tokens to wait for before generating the next target language token. meanwhile, @xcite proposed a simple wait-k method with fixed policy, where the decoder starts generating the target language tokens the moment k source language tokens are available. however, their model for simultaneous translation relies only on the source sentence. this research concentrates on the wait-k approach leveraging document-level information from previous context sentences. document-level nmt leverages context beyond the current sentence in order to improve translation quality @xcite @xcite @xcite @xcite . document-level nmt models can be implemented as a post-processing model or context-aware model. the post-processing models use an additional module to use context on generated translations @xcite . however, post-processing generated translations may lead to higher latency, which is counter-intuitive in a simultaneous translation scenario. on the other hand, context-aware models leverage additional context during translation. for example, @xcite proposed to simply concatenate the previous sentences in both the source and target side to the input to the system. @xcite al. (2017); bawden et al. (2018); zhang et al. (2018) use separate context encoder for a few previous source sentences. similarly, we also use a separate context encoder to extract document-level information. however, we incorporate document-level information into snmt in order to improve translation quality, where only information from the source sentence is insufficient during translation. ## experimental settings we describe experimental settings aimed at helping verify the degree to which document context helps improve translation quality in a simultaneous translation setting. ## results we describe the results of our experiments in resource-rich and resource-poor settings. ## analysis 6.1 translation of context-aware tokens we investigate whether snmt performance is improved by using contextual information. therefore, we created context-aware parallel data in which the target sentence contains the tokens related to the previous target sentence. for example, given the context source sentence "the 2008 taipei game show, organized by the taipei computer association (tca), ended on monday, and was different from shows of past years.", and the source sentence "this could be seen in the gaming population, industry, and exhibition arrangements." in a simultaneous manner, the generated target sentence should be "ゲーム の 人口 、 産業 、 そして 展示 会 の 配列 で 見る こと が でき た 。". here, "ゲーム" means "game" and it is a token related to the context. the context sentence contains information about the game, and this can help translate "ゲーム" that appears at the beginning of the target sentence, where it is not available yet from the source sentence (e.g., @xmath5. we randomly investigated such sentence pairs from the @xmath6. bush's talks with saudi leaders also are expected to cover arms sales. source before heading to saudi arabia, mr. bush visited dubai briefly. target サウジアラビア に 向かう 前 に 、 ブッシュ 氏 は ドバイ を 短 期間 訪問 し た 。 wait-2 w/o context 既に 割れ て いる 前 に 、 ブッシュ 氏 は ドバイ に つい て 言及 し た 。 (before it was already cracked, mr. bush mentioned dubai.) wait-8 w/o context サウジアラビア の 王室 に 証言 する 前 に 、 ブッシュ 氏 は ドバイ へ の 説明 を 訪問 し た 。 (before testifying before the saudi royal family, mr. bush visited dubai to explain.) full w/o context サウジアラビア の 王室 に 証言 する 前 に 、 ブッシュ 氏 は ドバイ へ の 説明 を 訪問 し た 。 (before testifying before the saudi royal family, mr. bush visited dubai to explain.) wait-2 w/ context サウジアラビア の イスラム 教 徒 の 前 に 、 ブッシュ 氏 は 先週 記者 を 訪問 し た 。 (before the saudi muslims, mr. bush visited the press last week.) wait-8 w/ context サウジアラビア の 王室 に 向 かっ て 前 に 、 ブッシュ 氏 は ドバイ を 訪問 し た 。 (before heading to the royal family in saudi arabia, mr. bush visited dubai.) full w/ context サウジアラビア の 王室 に 向 かっ て 前 に 、 ブッシュ 氏 は ドバイ を 訪問 し た 。 (before heading to the royal family in saudi arabia, mr. bush visited dubai.) figure 2 shows that bleu and accuracy results for contextual models, using up to one or two previous sentences 6 as context, for created context-aware parallel data. in bleu, it can be seen that the results are almost the same between the non-contextual and the contextual models. on the other hand, the results of accuracy differ between the non-contextual and the contextual models. in particular, accuracy is improved by considering the context at @xmath7, 2, and 4. from this result, it can be seen that tokens related to the context can be translated by considering the context in snmt. our analysis also leads us to believe that it is difficult for bleu to evaluate the improvement due to the context because bleu was not designed in that way. this shows that there is a need for context-aware evaluation mechanisms. ## conclusion we proposed wait-k document-level simultaneous nmt to complement the information of incomplete input during the translation process. our proposed method is to replace the source encoder and target language decoder with wait-k equivalents while keeping the context encoder. the experimental results show that the proposed method slightly improves the translation quality in high-resource settings but not by appreciable amounts in low-resource settings. the analysis showed that wait-k models are more context-aware and rely on context whenever it should be helpful. however, the current model is unable to successfully determine when the context should be used, preventing the successful utilization of context. this indicates that we need to investigate further more effective ways to utilize the previous sentences in the document as context. our human evaluation was also rather limited, and in the future, we plan to conduct a human evaluation to determine which kind of context-aware phenomena (pronoun disambiguation, word sense disambiguation) our approaches can address.