C o D o NMT : Modeling Cohesion Devices for Document-Level Neural Machine Translation (2022)

Abstract: Cohesion devices, e.g., reiteration, coreference, are crucial for building cohesion links across sentences. In this paper, we propose a document-level neural machine translation framework, CoDoNMT, which models cohesion devices from two perspectives: Cohesion Device Masking (CoDM) and Cohesion Attention Focusing (CoAF). In CoDM, we mask cohesion devices in the current sentence and force NMT to predict them with inter-sentential context information. A prediction task is also introduced to be jointly trained with NMT. In CoAF, we attempt to guide the model to pay exclusive attention to relevant cohesion devices in the context when translating cohesion devices in the current sentence. Such a cohesion attention focusing strategy is softly applied to the self-attention layer. Experiments on three benchmark datasets demonstrate that our approach outperforms state-of-the-art document-level neural machine translation baselines. Further linguistic evaluation validates the effectiveness of the proposed model in producing cohesive translations.

## introduction neural machine translation (nmt) has become the dominant approach for machine translation and achieved substantial progress in comparison to statistical machine translation. some studies even claim that nmt has reached human parity @xcite . despite this, most nmt models are at the sentence level, which translate documents sentence by sentence, ignoring inter-sentential dependencies. documents translated in this way are usually incoherent and inconsistent across sentences. in order to address this issue, a wide range of efforts have been made to leverage intersentential context information for document-level nmt @xcite @xcite @xcite @xcite . most efforts have been dedicated to modeling local or global context via additional encoders, attention, cache, concatenating inputs, etc @xcite . however, these approaches normally focus on the way of integrating context into translation, rather than the context itself. the basic assumption behind this is that models are able to detect relevant contextual information. however, @xcite and @xcite find that most of the improvements obtained by these approaches cannot be explained as leveraging the right context. we suggest that it could be not a good choice to treat contextual words equally and rely on models to learn contextual clues in an implicit way. this is because it is usually difficult for nmt to capture key information from a long context through itself @xcite . hence, pinpointing semantically or grammatically relevant context words in an explicit way is desirable for document-level nmt. in this paper, different from the aforementioned context modeling schemes, we model contextual information for document-level nmt in an explicit way via cohesion devices. cohesion devices (e.g., reiteration, co-reference) are widely acknowledged as important linguistic items that chain sentences into cohesive discourse @xcite . moreover, the interpretation of one cohesion device depends on the corresponding device that is paired to it @xcite . consider the following text "amy went to the party. she sat with sara.". the interpretation of the cohesion device she is deeply related to the cohesion device amy. therefore, explicitly modeling these cohesion devices may guide document-level nmt to actively explore contextual clues, so as to yield cohesive translations. inspired by this, we propose codonmt, as shown in fgiure 1, to explore cohesion devices for document-level neural machine translation. indicate the predicted words in translations of the context and the current sentence, respectively. particularly, we present cohesion device masking (codm) and cohesion attention focusing (coaf) as two essential components for codonmt, in an attempt to force nmt to explicitly predict masked cohesion devices and to focus its attention exclusively on related cohesion devices. ## cohesion devices a discourse is cohesive when sentences are properly linked by cohesion devices. from the linguistic perspective, cohesion devices can be divided into two categories: lexical cohesion devices and grammatical cohesion devices @xcite . in this paper, we consider reiteration, synonym and super-subordinate for lexical cohesion devices, and co-reference for grammatical cohesion devices. we choose these devices because they are common and can be annotated automatically. reiteration: reiteration refers to the repetition of the same words in a discourse. this is a common phenomenon in discourse @xcite . and it is easy to detect. note that we exclude stop words when detecting any type of cohesion devices. synonym: we use wordnet @xcite to define synonyms, which is a large lexical database of english. nouns, verbs, adjectives and adverbs are grouped into sets of semantic groups called synsets. we denote synset(w) as a set that includes synonyms grouped in the same synset as word w in wordnet. super-subordinate: superordinate and subordinate are formed by words with an is-a semantic relationship, such as apple and fruit (hypernym), furniture and cupboard (hyponym), and so on. as the super-subordinate relation is also encoded in wordnet, we still use wordnet to detect hypernyms and hyponyms. let hypset(w) be a set that includes both hypernyms and hyponyms in word-net for word w. co-reference: co-reference is a relationship between two words or phrases in which both refer to the same person or thing and one is a linguistic antecedent of the other. we use corenlp @xcite to parse co-reference relations between the current sentence and its context sentences. ## experiments to examine the effectiveness of our proposed approaches, we carried out experiments on three widely-used datasets and linguistic evaluation on a contrastive test set. ## analysis in this section, we analyzed three factors to examine their impact on the performance of the proposed model, including: 1) cohesion device type, 2) masking strategy, 3) cohesion attention mask. ## related work document-level nmt aims to improve translation quality with the aid of contextual information beyond the scope of current sentences. most previous works focus on the integration of the intersentential context into nmt. one typical approach is introducing an addtional encoder to encode context, and then integrate the context representation into the primary encoder and/or the decoder @xcite @xcite @xcite . miculicich et al. (2018) propose a hierarchical attention model to capture the contextual information from both word and sentence level. tan et al. (2019) propose a hierarchical model to learn global context for document-level nmt. voita et al. (2019) introduce a two-pass framework that uses several previous sentences as context to refine the translation generated by a strong sentence-level nmt. zhang et al. (2021) apply a multi-hop mechanism to documentlevel nmt to simulate the human-like draft-editing and reasoning process. liu et al. (2020), ma et al. (2020) and bao et al. (2021) combine pre-trained models with document-level nmt. zhang et al. (2020) pretrain a source context prediction model on a large-scale monolingual document corpus to learn contextualized sentence embeddings. yet another research strand is to focus on context selection. maruf and haffari (2018), kuang et al. source c x c x c x x like your boss said, might get you killed. well, that's what i keep hearing. nobody wants to share this dangerous entity's idea with me. maybe you know. reference c x c x c x x kak skazal tvo boss, to moet konqit~s tvoe smert~. qto , to slyxu vse vrem. nikto ne hoqet delit~s so mno liqnost~ to opasno suwnosti. moet, ty znaex~. ## conclusion in this paper, we have presented codonmt for document-level nmt, which models cohesion devices with two key methods, codm and coaf. codm masks cohesion devices in the current sentence to force the model to actively explore intersentential contextual information. coaf softly guides the model to focus attention on cohesion devices. both automatic and linguistic evaluations show that our model can significantly im-prove translation quality in terms of bleu and lexical and grammatical cohesion accuracy on a discourse-oriented contrastive test set. further analyses demonstrate the impact of cohesion device type and masking strategy on translation quality.