Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021)

Abstract: Recently a number of approaches have been proposed to improve translation performance for document-level neural machine translation (NMT). However, few are focusing on the subject of lexical translation consistency. In this paper we apply “one translation per discourse” in NMT, and aim to encourage lexical translation consistency for document-level NMT. This is done by first obtaining a word link for each source word in a document, which tells the positions where the source word appears. Then we encourage the translation of those words within a link to be consistent in two ways. On the one hand, when encoding sentences within a document we properly share context information of those words. On the other hand, we propose an auxiliary loss function to better constrain that their translation should be consistent. Experimental results on Chinese↔English and English→French translation tasks show that our approach not only achieves state-of-the-art performance in BLEU scores, but also greatly improves lexical consistency in translation.

## introduction unlike sentence-level neural machine translation (nmt), document-level nmt needs to not only model intra-sentence dependencies, but also consider a wide variety of inter-sentence discourse phenomena, such as coreference, lexical cohesion, semantic coherence, discourse relations. motivated by the success of "one translation per discourse" in statistical machine translation (smt) @xcite @xcite @xcite , in this paper our goal is to encourage lexical translation consistency for document-level nmt. figure 1 shows an example of an input document and its output translated by a state-of-the-art sentence-level nmt system. the technical term * corresponding author: junhui li. 房地产业/fang_di_chan_ye, occurring four times within a document, surprisingly obtains different translations while in its reference (human translation) it is translated consistently. such inconsistent translations, however, tend to confuse readers in some cases. recent years have witnessed an increasing interest in document-level nmt, but most previous studies explore various context-aware models for better incorporating document-level context to improve translation performance without handling a specific discourse phenomenon @xcite @xcite , to name a few). as a way to encourage lexical translation consistency, @xcite and @xcite cache recently translated words and/or their translations for translating future sentences. however, cache-based approaches may potentially guide the translation of future sentences in a wrong way since the cached translation could be incorrect. rather than explicitly presenting lexical translations used in previous sentences as in cache-based approaches, in this paper we aim at improving lexical translation consistency in a softer way: we encourage translations of the same word in different positions of a document to be consistent. specifically, we first obtain a word link for each source word in a document if it has, which tells the positions the source word appears at. to encourage translation consistency for words within a link, we exchange their context information when encoding sentences in a document. moreover, we properly propose an auxiliary loss function to better constrain that the translations of these words should be consistent. overall, we make the following contributions. one nice property of our approach is that our models could synchronously translate sentences in a document, rather than translating them one by one as in cached-based approaches. ## motivation given a parallel document pair (s, t ), a sourceside word w (stemmed to eliminate morphological differences if necessary) is one of words of our interest if it is a non-stop word and occurs two or more times in s. for w, we conjecture that the translations (stemmed too if necessary) of w in t tend to be same. as shown in figure 1 , source word 房地产业/fang_di_chan_ye is consistently translated into (the) real estate sector in reference translation. ## encouraging lexical translation consistency via word links as our goal is to encourage lexical consistency in document-level translation, we first obtain word links, each of which tells the positions that a word appears in a document (section 3.1). to encourage translation consistency among words in the same link, on the one hand we exchange their information when encoding sentences within a document (section 3.2). on the other hand, we properly propose an auxiliary loss function to better constrain the translations of these words being consistent (section 3.3). ## discussion next, we take @xmath0. see appendix for more discussion. ## related work there has been substantial work in smt that either encourages or enforces lexical translation consistency. for example, @xcite and @xcite propose post-editing approaches to re-translate those source words which have been translated differently in a document. @xcite and @xcite propose cache-based approaches to remember translation history. discriminative learning approaches @xcite are also proposed to fix lexical translation non-consistency. besides, @xcite and @xcite demonstrate that applying "one translation per discourse" constraint in smt leads to better translation quality. moving to nmt, most of document-level nmt studies have proposed various context-aware nmt models to leverage either local context, e.g., previous sentences @xcite @xcite @xcite @xcite , or entire document @xcite @xcite @xcite @xcite . however, different from ours, these studies aim to improve the translation accuracy without handling a specific discourse phenomena. @xcite and @xcite cache recently translated words and/or their translations which could be used to increase lexical consistency when translate future sentences. however, cache-based approaches require to translate sentences in a document one by one and may potentially guide the translation of future sentences in a wrong way since the cached translations could be incorrect. experimental re-sults in related studies @xcite have shown that the improvement of cache-based approaches is limited in bleu over (sentence-level) transformer. our approach is different from cached-based approach as we translate sentences within a document synchronously, and more importantly it does not explicitly suggest any translation. there also exists many studies in nmt that aim to resolve discourse phenomena in post-process. for example, to make translation outputs of a document more coherent, @xcite propose docrepair trained on monolingual target language documents to correct the inconsistencies in sentence-level translation while @xcite train a context-aware language model to re-rank sentence-level translation candidates. ## conclusion in this paper, we apply "one translation per discourse" in nmt, and have proposed an approach to encourage lexical translation consistency. this is done by first obtaining a word link for each source word in a document, which tells the positions the source word appears at. then we encourage the translations of words within a link to be consistent by both exchanging their context information in encoding, and using an auxiliary loss to constrain their translation being consistent. experimental results on chinese↔english and @xmath1, but also greatly improves lexical translation consistency.