Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning (2020)

Abstract: Document-level neural machine translation has yielded attractive improvements. However, majority of existing methods roughly use all context sentences in a fixed scope. They neglect the fact that different source sentences need different sizes of context. To address this problem, we propose an effective approach to select dynamic context so that the document-level translation model can utilize the more useful selected context sentences to produce better translations. Specifically, we introduce a selection module that is independent of the translation module to score each candidate context sentence. Then, we propose two strategies to explicitly select a variable number of context sentences and feed them into the translation module. We train the two modules end-to-end via reinforcement learning. A novel reward is proposed to encourage the selection and utilization of dynamic context sentences. Experiments demonstrate that our approach can select adaptive context sentences for different source sentences, and significantly improves the performance of document-level translation methods.

## introduction although neural machine translation (nmt) has achieved great progress in recent years @xcite @xcite , when fed an entire document, standard nmt systems translate sentences in isolation without considering the cross-sentence dependencies. consequently, document-level neural machine translation (docnmt) methods are proposed to utilize source-side or target-side intersentence contextual information to improve translation quality over sentences in a document @xcite @xcite @xcite @xcite . more recently, researchers of docnmt mainly focus on exploring various attention-based networks to leverage the cross-sentence context efficiently, and evaluate the special discourse phenomena @xcite @xcite . however, there is still an issue that has received less attention: which context sentences should be used when translating a source sentence? we conduct an experiment to verify an intuition: the translation of different source sentences requires different context. as shown in majority of existing docnmt models set the context size or scope to be fixed. they utilize all of the previous k context sentences @xcite @xcite @xcite , or the full context in the entire document @xcite @xcite . as a result, the inadequacy or redundancy of contextual information is almost inevitable. from this viewpoint, @xcite propose a selective attention approach that uses the sparsemax function @xcite instead of the softmax to normalize the attention weights. the sparsemax assigns the low probability in softmax to zero so that the model can focus on the sentences with high probability. however, the learning of attention weights lacks guidance, and they cannot handle the situation where the source sentences achieve the best translation results without relying on any context, which happens in about 39.4% of sentences in the experiment. to address the problem, we propose an effective approach to select contextual sentences dynamically for each source sentence in the documentlevel translation. specifically, we propose a context scorer to score each candidate context sentence according to the currently translated source sentence. then, we utilize two selection strategies to select useful context sentences for the translation module. the size of selected context is variable for different sentences. a core challenge of our approach is that the selection process is nondifferentiable. therefore, we leverage the reinforcement learning (rl) method to train the selection and docnmt modules together. we design a novel reward to encourage the model to be aware of different context sentences and select more appropriate context to improve translation quality. in this paper, we make the following contributions: ## document-level machine translation a standard docnmt system generally translates a source sentence @xmath0, • • • , x i } to a target sentence @xmath1, • • • , y t } with the aid of contextual information z that is usually a subset of the candidate context set z. the model is trained to minimize the negative log-likelihood as: different granularity (word or sentence) and different sources (source-side or target-side) of contextual information z have been explored. @xcite divide the candidate context set z into two cases: offline where the context comes from the entire document, and online that only uses the past context. in this paper, we mainly focus on a general scenario, where docnmt translates sentences with the online source-side context sentences. ## dynamic context selection our approach translates a source sentence x in the document in two steps. first, we select the appropriate context sentences for the translation of x via the selection module. independent of doc-nmt module, this step is conducted before the context encoding in docnmt module. the core component is a context scorer that calculates the contribution of each context sentence @xmath2.1). according to the context scores, we propose two strategies to choose the useful context sentences (sub-section 3.2). sec-ond, we feed the selected context sentences into a docnmt module to generate the translation. to overcome the non-differentiable behavior of the context selection and the lack of direct supervision when training the context scorer, we connect the two steps through the reinforcement learning strategy. we propose an effective reward that is related to the translation quality to guide the dynamic selection of context sentences and the optimization of parameters in docnmt model (sub-section 3.3). ## related work standard neural machine translation methods usually focus on the sentence-level translation @xcite @xcite @xcite @xcite @xcite . as a contrast, document-level neural machine translation methods mainly pay attention to how to utilize the cross-sentence context. researchers propose various context-aware networks to utilize contextual information to improve the performance of docnmt models on the translation quality @xcite @xcite or discourse phenomena @xcite @xcite . however, most methods roughly leverage all context sentences in a fixed size that is tuned on development sets @xcite @xcite @xcite , or full context in the entire document @xcite @xcite . they ignore the individualized needs for context when translating different source sentences. some works have noticed that not all context is useful @xcite . @xcite explore the context selection in the single-encoder framework @xcite , and select context sentences that yield highest forced back-translation probability. however, the method cannot optimize doc-nmt model at training phase, and requires backtranslation model at inference phrase. @xcite sharpen the attention weights between the source and context sentences through the sparse-max function, and implicitly select context with high attention weights. nevertheless, the method lacks direct supervision over context selection, and it cannot cover the situation where context is not needed. inspired by the extractive-abstractive summarization @xcite , our approach is different from above docnmt methods. our approach can explicitly select dynamic size (that can be 0) of context sentences for the translation of different source sentences. ## conclusion and future work we propose a dynamic selection method to choose variable sizes of context sentences for documentlevel translation. the candidate context sentences are scored and selected by two proposed strategies. we train the whole model via reinforcement learning, and design a novel reward to encourage the selection of useful context sentences. when applied to existing docnmt models, our approach can improve translation quality significantly. in the future, we will select context sentences in larger candidate space, and explore more effective ways to extend our approach to select target-side context sentences.