Capturing document context inside sentence-level neural machine translation models with self-training (2021)

Abstract: Neural machine translation (NMT) has arguably achieved human level parity when trained and evaluated at the sentence-level. Document-level neural machine translation has received less attention and lags behind its sentence-level counterpart. The majority of the proposed document-level approaches investigate ways of conditioning the model on several source or target sentences to capture document context. These approaches require training a specialized NMT model from scratch on parallel document-level corpora. We propose an approach that doesn’t require training a specialized model on parallel document-level corpora and is applied to a trained sentence-level NMT model at decoding time. We process the document from left to right multiple times and self-train the sentence-level model on pairs of source sentences and generated translations. Our approach reinforces the choices made by the model, thus making it more likely that the same choices will be made in other sentences in the document. We evaluate our approach on three document-level datasets: NIST Chinese-English, WMT19 Chinese-English and OpenSubtitles English-Russian. We demonstrate that our approach has higher BLEU score and higher human preference than the baseline. Qualitative analysis of our approach shows that choices made by model are consistent across the document.

## introduction neural machine translation (nmt) @xcite @xcite has achieved great success, arguably reaching the levels of human parity @xcite on chinese to english news translation that led to its popularity and adoption in academia and industry. these models are predominantly trained and evaluated on sentence-level parallel corpora. document-level machine translation that requires capturing the context to accurately translate sentences has been recently gaining more popularity and was selected as one of the main tasks in the premier machine translation conference wmt19 @xcite and wmt20 @xcite . a straightforward solution to translate documents by translating sentences in isolation leads to inconsistent but syntactically valid text. the inconsistency is the result of the model not being able to resolve ambiguity with consistent choices across the document. for example, the recent nmt system that achieved human parity @xcite inconsistently used three different names "twitter move car", "wechat mobile", "wechat move" when referring to the same entity @xcite . to tackle this issue, the majority of the previous approaches @xcite @xcite @xcite @xcite @xcite @xcite proposed contextconditional nmt models trained on documentlevel data. however, none of the previous approaches are able to exploit trained nmt models on sentence-level parallel corpora and require training specialized context-conditional nmt models for document-level machine translation. we propose a way of incorporating context into a trained sentence-level neural machine translation model at decoding time. we process each document monotonically from left to right one sentence at a time and self-train the sentence-level nmt model on its own generated translation. this procedure reinforces choices made by the model and hence increases the chance of making the same choices in the remaining sentences in the document. our approach does not require training a separate context-conditional model on parallel document-algorithm 1: document-level nmt with self-training at decoding time input: document @xmath0, ..., x n ), pretrained sentence-level nmt model f (θ), learning rate α, decay prior λ and number of passes over document p output: we make the key contribution in the paper by introducing the document-level neural machine translation approach that does not require training a context-conditional model on document data and does not require separate document-level language model to rank the outputs of the nmt model according to consistency of translated document. we show how to adapt a trained sentence-level neural machine translation model to capture context in the document during decoding. we evaluate and demonstrate improvements of our proposed approach measured by bleu score and preferences of human annotators on several document-level machine translation tasks including nist chinese-english, wmt19 chinese-english and opensubtitles english-russian datasets. we qualitatively analyze the decoded sentences produced using our approach and show that they indeed capture the context. ## proposed approach we translate a document d consisting of n source sentences x 1 , x 2 , ..., x n into the target language, given a well-trained sentence-level neural machine translation model f θ . the sentencelevel model parametrizes a conditional distribution p(y |x) = t @xmath1, x) of each target word y t given the preceding words @xmath2. decoding is done by approximately finding arg max y p(y |x) using greedy decoding or beam-search. f is typically a recurrent neural network with attention @xcite or a transformer model @xcite with parameters θ. ## related work although there have been some attempts at tackling document-level neural machine translation (for example see proceedings of discourse in machine translation workshop (popescu-belis et al., 2019)), it has largely received less attention compared to sentence-level neural machine translation. prior document-level nmt approaches @xcite @xcite @xcite @xcite proposed different ways of conditioning nmt models on several source sentences in the document. perhaps closest of those document nmt approaches to our work is the approach by @xcite , where they train a nmt model with a separate non-parametric cache @xcite ) that incorporates topic information about the document. recent approaches @xcite @xcite use only partially available parallel document data or monolingual document data. these approaches proposed to fill in missing context in the documents with random or generated sentences. another line of document-level nmt work @xcite proposed a twopass document decoding model inspired by the deliberation network @xcite in order to incorporate target side document context. a parallel line of work @xcite @xcite introduced document-level approaches that do not require training the context-conditional nmt model by introducing a separate language model to enforce the consistency in the outputs of sentence-level nmt model. @xcite used a simple n-gram based semantic space language model @xcite to re-rank the outputs of the sentence-level nmt model inside the beam-search algorithm to enforce documentlevel consistency. @xcite proposed a novel beam search method that incorporates document context inside noisy channel model @xcite @xcite that uses a powerful gpt2 language model @xcite . similar to our work, their approach doesn't require training context-conditional models on parallel document corpora, but relies on separate target-to-source nmt model and unconditional language model to re-rank hypotheses of the sourceto-target nmt model. closest to our work is the dynamic evaluation approach proposed by @xcite and further extended by graves (2013); @xcite , where a neural language model is trained at evaluation time. however unlike language modeling where inputs are ground-truth targets used both during training and evaluation, in machine translation ground-truth translation are not available at decoding time in practical settings. the general idea of storing memories in the weights of the neural network rather than storing memories as copies of neural network activations, that is behind our approach and dynamic evaluation, goes back to 1970s and 1980s work on associative memory models @xcite @xcite and to more recent work on fast weights @xcite . our work belongs to the broad category of selftraining or pseudo-labelling approaches @xcite proposed to annotate the unlabeled data to train supervised classifiers. selftraining has been successfully applied to nlp tasks such as word-sense disambiguation @xcite and parsing @xcite @xcite . self-training has also been used to label monolingual data to improve the performance of sentencelevel statistical and neural machine translation models @xcite . recently, @xcite proposed noisy version of self-training and showed improvement over classical self-training on machine translation and text summarization tasks. backtranslation @xcite is another popular pseudo-labelling technique that utilizes target-side monolingual data to improve performance of nmt models. @xcite @xcite . the last four rows show performance of our baseline sentence-level transformer models with and without self-training. bt: backtranslation. ## results we present translation quality results measured by bleu on nist dataset on in the results on the wmt19 evaluation sets are presented on ## human evaluation we conduct a human evaluation study on the nist zh-en and the opensubtitles en-ru datasets. for both datasets we sample 50 documents from the test set where translated documents generated by the self-training approach are not exact copies of the translated documents generated by the sentencelevel baseline model. for the nist zh-en dataset we present reference documents, translated documents generated by the sentence-level baseline, and translated documents generated by self-training approach to 4 native english speakers. for the open-subtitles en-ru dataset we follow a similar setup, where we present reference documents, translated documents generated by sentence-level baseline, and translated documents generated by self-training approach to 4 native russian speakers. all translated documents are presented in random order with no indication of which approach was used to generate them. we highlight the differences between translated documents when presenting them to human evaluators. the human evaluators are asked to pick one of two translations as their preferred option for each document. we ask the human evaluators to consider fluency, idiomaticity and correctness of the translation relative to the reference when entering their preferred choices. we follow the setup of @xcite when performing human evaluation. we collect a total of 200 annotations for 50 documents from all 4 human evaluators and show results in ## qualitative results in in the first two examples we emphasize the gender of the person marked on verbs and adjectives in translated russian sentences. in the first example, the baseline sentence-level model inconsistenly produces different gender markings on the underlined verb сказал (masculine told) and underlined adjective сильной (feminine strong). the selftraining approach correctly generates a translation with consistent male gender markings on both the underlined verb сказал and the underlined adjective сильным. similarly, in the second example, the baseline model inconsistenly produces different gender markings on the underlined verbs приглашена (feminine invited) and поругался (masculine fought). self-training consistently generates female gender markings on both the underlined verbs приглашена (feminine invited) and поссорилась (feminine fought). in the third example, we emphasize the underlined named entity in reference and generated translations. the baseline sentence-level model inconsistently generates the names "doyle" and "du" when referring to the same entity across two sentences in the same document. the self-training approach consistently uses the name "doyle" across two sentences when referring to the same entity. in the fourth example, we emphasize the plurality of the underlined words. the baseline model inconsistenly generates both singular and plural forms when referring to same noun in consecutive sentences. self-training generates the noun "pilots" in correct plural form in both sentences. ## conclusion in this paper, we propose a way of incorporating the document context inside a trained sentencelevel neural machine translation model using selftraining. we process documents from left to right multiple times and self-train the sentence-level nmt model on the pair of source sentence and generated target sentence. this reinforces the choices made by the nmt model thus making it more likely that the choices will be repeated in the rest of the document. we demonstrate the feasibility of our approach on three machine translation datasets: nist zh-en, wmt19 zh-en and opensubtitles en-ru. we show that self-training improves sentence-level ref мы с эйприл развелись . как я и сказал ... игра в ожидание . будь сильным . и всё получится . baseline мы с эйприл развелись . ну , как я уже сказал ... игра ожидания . будь сильной . ты справишься . ours мы с эйприл развелись . ну , как я уже сказал ... игра ожидания . будь сильным . ты справишься . ref сёрен устраивает вечеринку по поводу своего дня рождения в субботу , а я не знаю , пойду ли я . почему бы тебе не пойти ? просто всё пошло не так . -и я поссорился с кнудом . baseline в субботу день рождения сёрена и я не знаю , приглашена ли я . почему тебя не пригласили ? все просто пошло не так . -и я поругался с кнудом . ours в субботу день рождения сёрена и я не знаю , приглашена ли я . почему тебя не пригласили ? все просто пошло не так . -и я поссорилась с кнудом . ref we are actively seeking a local partner to set up a joint fund company , " duchateau said . duchateau said that the chinese market still has ample potentials . baseline we are actively looking for a local partner to establish a joint venture fund company , " doyle said . ref in may this year , 13 pilots with china eastern airlines wuhan company in succession handed in their resignations , which were rejected by the company . soon afterwards , the pilots applied one after another at the beginning of june to the labor dispute arbitration commission of hubei province for labor arbitration , requesting for a ruling that their labor relationship with china eastern airlines wuhan company be terminated . baseline in may this year , 13 pilots of china eastern 's wuhan company submitted their resignations one after another , but the company refused . the pilot then applied for labor arbitration with the hubei province labor dispute arbitration committee in early june , requesting the ruling to terminate the labor relationship with the wuhan company of china eastern airlines . ours in may this year , 13 pilots of china eastern 's wuhan company submitted their resignations one after another , but the company refused . subsequently , in early june , the pilots successively applied for labor arbitration with the hubei province labor dispute arbitration committee , requesting that the labor relationship with china eastern airlines be terminated . in this work, we only use self-training on sourceto-target nmt models in order to capture the target side document context. one extension could investigate the application of self-training on both target-to-source and source-to-target sentence-level models to incorporate both source and target document context into generated translations. overall, we hope that our work would motivate novel approaches of making trained sentence-level models better suited for document translation at decoding time.