Modeling Consistency Preference via Lexical Chains for Document-level Neural Machine Translation (2022)

Abstract: In this paper we aim to relieve the issue of lexical translation inconsistency for document-level neural machine translation (NMT) by modeling consistency preference for lexical chains, which consist of repeated words in a source-side document and provide a representation of the lexical consistency structure of the document. Specifically, we first propose lexical-consistency attention to capture consistency context among words in the same lexical chains. Then for each lexical chain we define and learn a consistency-tailored latent variable, which will guide the translation of corresponding sentences to enhance lexical translation consistency. Experimental results on Chinese→English and French→English document-level translation tasks show that our approach not only significantly improves translation performance in BLEU, but also substantially alleviates the problem of the lexical translation inconsistency.

## introduction generally, the translations of source-side words repeated within a document tend to be consistent @xcite @xcite @xcite @xcite while sentencelevel neural machine translation (nmt) suffers from the serious problem of lexical translation inconsistency due to the lack of inter-sentence context. although most recent studies in documentlevel nmt propose various context-aware models to better capture inter-sentence context, they do not handle specific discourse phenomena, e.g., lexical consistency. in this paper, we therefore study lexical consistency for document-level nmt by modeling consistency preference for lexical chains which represent the lexical consistency structure of a text. lexical translation consistency is a common discourse phenomenon. many studies in statistical machine translation (smt) @xcite @xcite @xcite discuss and apply the one translation per discourse hypothesis. more recently, @xcite give a detailed analysis about the lexical translation consistency on @xmath0. in their analysis, the proportion of words related with this phenomenon reaches about 20 percent against all of words in the whole corpus. furthermore, they find out that the translations of the words repeated within a document indeed tend to be consistent. intuitively, enhancing lexical translation consistency should consider inter-sentence context. however, existing researches in document-level nmt mainly focus on capturing inter-sentence context in general and do not explicitly handle specific discourse phenomena. as a result, these models have limited effect on enhancing lexical translation consistency. different from them, both @xcite and @xcite recently introduce auxiliary consistency losses to encourage the translations of the repeated source-side words being same. 1 alternatively, in this paper we propose a softer approach to model consistency preference for lexical chains in document-level nmt, where the lexical chains consist of repeated words in a sourceside document. specifically, we enhance the translation consistency of lexical chains from two aspects: 1) we propose a lexical-consistency attention to capture consistency context among words in the same lexical chains while we also use a generalcontext attention to capture general inter-sentence context; and 2) we propose a consistency-tailored latent variable for each lexical chain to model its consistency preference, which will guide the translation of the lexical chain in decoding. these latent variables are properly learned via a conditional variational autoencoder (cvae) module, which does not explicitly constrain the translations of repeated source words to be same, and thus could ease over-correction. experimental results on chineseto-english and french-to-english document-level translation tasks show that our approach not only significantly improves the translation performance in bleu, but also greatly alleviates the problem of lexical translation inconsistency. ## related work there have been substantial studies in documentlevel nmt that focus on effective utilization of general inter-sentence context. these studies can be roughly categorized into three groups, including those who only consider the source-side intersentence context @xcite @xcite @xcite @xcite @xcite , those who only consider the target-side inter-sentence context (or translation history) @xcite @xcite , and those who consider both the source and target inter-sentence context @xcite @xcite @xcite . different from ours, these studies leverage the inter-sentence context in a indistinguishable way and do not handle specific discourse phenomena. therefore, although they achieve impressive improvement of translation accuracy with the expectation of alleviating the discourse issues in general, they usually have limited effect on enhancing lexical translation consistency. there also exist many researches in mt that aim to enforce or encourage lexical translation consistency. in smt, @xcite , @xcite and @xcite propose post-editing approaches to enforce lexical translation consistency by re-translating those repeated source words which have been translated differently. @xcite and @xcite propose cachebased approaches to encourage translation consistency. @xcite and @xcite propose discriminative approaches to improve the consistency of translations. @xcite add three super-sentential "consistency features" to the trans-lation model. beside, @xcite propose to improve the translation consistency of repeated nouns in post-editing or/and reranking. in nmt, @xcite and @xcite propose consistency losses which encourage the translations of repeated words being same. different from theirs, our approach automatically models the translation consistency preference of repeated words via a cvae module in a softer way without explicitly constraining their translations to be same. ## proposed approach as our goal is to model consistency preference for lexical chains, we first construct lexical chains of repeated words in the source-side document. each lexical chain represents a repeated word that appear two or more times in the document. then we encode source-side documents with prepared lexical chains by a consistency-aware encoder (section 3.1). meanwhile, we propose to learn a consistency-tailored latent variable for each lexical chain (section 3.2) by a cvae-based module. these learned latent variables are dynamically integrated into each decoding step to further enhance translation consistency (section3.3). finally, we define a joint training objective to optimize the model with the cvae-based module (section 3.4). we define some notations before describing our approach. given a parallel document pair (x , y) = {xi, yi}| n @xmath1, we assume that each source sentence xi consists of n words ## experimentation as inspired by the conclusion in @xcite and @xcite that lexical translation consistency is encouraged in chinese (zh)→english (en) and french (fr)→english (en) human translation, we evaluate our approach on {zh, fr}→en document-level translation tasks. ## discussion next, we take @xmath2. we also provide more discussion in appendix d∼g. ## conclusion in this paper, we have proposed a lexical-chain based approach to alleviate the issue of translation inconsistency for document-level nmt. we first use lexical-consistency attention to capture consistency context among words in the same lexical chains. then we learn a consistency-tailored latent variable for each lexical chain to model consistency preference in translation. experimental results on @xmath4.