Leveraging Discourse Rewards for Document-Level Neural Machine Translation (2020)

Abstract: Document-level machine translation focuses on the translation of entire documents from a source to a target language. It is widely regarded as a challenging task since the translation of the individual sentences in the document needs to retain aspects of the discourse at document level. However, document-level translation models are usually not trained to explicitly ensure discourse quality. Therefore, in this paper we propose a training approach that explicitly optimizes two established discourse metrics, lexical cohesion and coherence, by using a reinforcement learning objective. Experiments over four different language pairs and three translation domains have shown that our training approach has been able to achieve more cohesive and coherent document translations than other competitive approaches, yet without compromising the faithfulness to the reference translation. In the case of the Zh-En language pair, our method has achieved an improvement of 2.46 percentage points (pp) in LC and 1.17 pp in COH over the runner-up, while at the same time improving 0.63 pp in BLEU score and 0.47 pp in F-BERT.

## introduction the recent advances in neural machine translation (nmt) @xcite @xcite have provided the research community and the commercial landscape with effective translation models that can at times achieve near-human performance. however, this usually holds at phrase or sentence level. when using these models in larger units of text, such as paragraphs or documents, the quality of the translation may drop considerably in terms of discourse attributes such as lexical and stylistic consistency. in fact, document-level translation is still a very open and challenging problem. the sentences that make up a document are not unrelated pieces of text that can be predicted independently; rather, a set of sequences linked together by complex underlying linguistics aspects, also known as the discourse @xcite . the discourse of a document includes several properties such as grammatical cohesion @xcite , lexical cohesion (halliday and hasan, 2014), document coherence @xcite and the use of discourse connectives @xcite . ensuring that the translation retain such linguistic properties is expected to significantly improve its overall readability and flow. however, due to the limitations of current decoder technology, nmt models are still bound to translate at sentence level. in order to capture the discourse properties of the source document in the translation, researchers have attempted to incorporate more contextual information from surrounding sentences. most document-level nmt approaches augment the model with multiple encoders, extra attention layers and memory caches to encode the surrounding sentences, and leave the model to implicitly learn the discourse attributes by simply minimizing a conventional nll objective. the hope is that the model will spontaneously identify and retain the discourse patterns within the source document. conversely, very little work has attempted to model the discourse attributes explicitly. even the evaluation metrics typically used in translation such as bleu @xcite are not designed to assess the discourse quality of the translated documents. for these reasons, in this paper we propose training an nmt model by directly targeting two specific discourse metrics: lexical cohesion (lc) and coherence (coh). lc is a measure of the frequency of semantically-similar words co-occurring in a document (or block of sentences) @xcite . for example, car, vehicle, engine or wheels are all semantically-related terms. there is significant empirical evidence that ensuring lexical cohesion in a text eases its understanding @xcite . at its turn, coh measures how well adjacent sentences in a text are linked to each other. in the following example from @xcite : "john took a train from paris to istanbul. he likes spinach." the two sentences make little 'sense' one after another. an incoherent text, even if grammatically and syntactically perfect, is anecdotally very difficult to understand and therefore coherence should be actively pursued. relevant to translation, @xcite has found that a high percentage of the human post-editing changes over machine-generated translations involves the improvement of cohesion and coherence. several lc and coh metrics that well correlate with the human judgement have been proposed in the literature. however, like bleu and most other evaluation metrics, they are discrete, non-differentiable functions of the model's parameters. hereafter, we propose to overcome this limitation by using the well-established policy gradient approach from reinforcement learning @xcite which allows using any evaluation metric as a reward without having to differentiate it. by combining different types of rewards, the model can be trained to simultaneously achieve more lexicallycohesive and more coherent document translations, while at the same time retaining faithfulness to the reference translation. ## baseline models this section describes the baseline nmt models used in the experiments. in detail, subsection 3.1 recaps the standard sentence-level translation model while subsection 3.2 describes the recent, strong hierarchical baseline that we have augmented with discourse rewards. ## risk training with discourse rewards in order to improve the baseline models, we propose to use the lc @xcite and coh @xcite evaluation metrics as rewards during training, so that the model is explicitly rewarded for generating more cohesive and coherent translation at document level. for that, we use a reinforcement learning approach, which allows using discrete, non-differentiable functions as rewards in the objective. following @xcite , we have used the structured loss that achieved the best results in their experiments, namely the expected risk minimization (risk) objective: by minimizing this risk objective, the nmt model is encouraged to give higher probability to candidate translations that obtain a higher reward. this function has been used at sentence level by @xcite . however, the same metrics could also be computed at document level by simply concatenating all the sentences from the same document together (both for the ground truth and the predictions). as a result, m now would be the number of words in a document, u(x) the candidate document translations, x the source document and y the reference document. computing the risk objective in this way permits having document-level reward functions as r(•). ## conclusion in this paper, we have presented a novel training method for document-level nmt models that uses discourse rewards to encourage the models to generate more lexically cohesive and coherent translations at document level. as training objective we have used a reinforcement learning-style function, named risk, that permits using discrete, non-differentiable terms in the objective. our results on four different language pairs and three translation domains have shown that our models have achieved a consistent improvement in discourse metrics such as lc and coh, while retaining comparable values of accuracy metrics such as bleu and f bert . in fact, on certain datasets, the models have even improved on those metrics. while the approach has proved effective in most cases, the best combination of discourse rewards, accuracy rewards and nll has had to be selected by validation for each dataset. in the near future we plan to investigate how to automate this selection, and also explore the applicability of the proposed approach to other natural language generation tasks.