Modeling Discourse Structure for Document-level Neural Machine Translation (2020)

Abstract: Recently, document-level neural machine translation (NMT) has become a hot topic in the community of machine translation. Despite its success, most of existing studies ignored the discourse structure information of the input document to be translated, which has shown effective in other tasks. In this paper, we propose to improve document-level NMT with the aid of discourse structure information. Our encoder is based on a hierarchical attention network (HAN) (Miculicich et al., 2018). Specifically, we first parse the input document to obtain its discourse structure. Then, we introduce a Transformer-based path encoder to embed the discourse structure information of each word. Finally, we combine the discourse structure information with the word embedding before it is fed into the encoder. Experimental results on the English-to-German dataset show that our model can significantly outperform both Transformer and Transformer+HAN.

## introduction neural machine translation (nmt) has made great progress in the past decade. in practical applications, the need for nmt systems has expanded from individual sentences to complete documents. therefore, document-level nmt has gradually drawn much more attention. contextual information is particularly important for obtaining highquality document translation. to get better contextual information, researchers have proposed many methods (e.g., memory network and hierarchical attention network) for document-level translation @xcite @xcite @xcite @xcite @xcite @xcite . discourse structure, as well as raw contextual sentences, is a major component of the document. and it has been proved to be effective in many other tasks, such as automatic document summarization @xcite and sentiment classification @xcite . however, to the best of our knowledge, discourse structure has not been explicitly used in document-level nmt. to address the above problem, we propose to improve document-level nmt with the aid of discourse structure information. first, we represent each input document with a rhetorical structure theory-based discourse tree @xcite . then, we use a transformer-based path encoder to embed the discourse structure path of each word and combine it with the corresponding word embedding before feeding it into the sentence encoder. in this way, discourse structure information can be fully exploited to enrich word representations and guide the context encoder to capture the relevant context of the current sentence. finally, we adopt han @xcite as our context encoder to model context information in a hierarchical manner. our contributions are as follows: (i) we propose a novel and efficient approach to explicitly exploit discourse structure information for documentlevel nmt. particularly, our approach is applicable for any other context encoder of document-level nmt; (ii) we carry out experiments on english-to-german translation task and experimental results show that our model outperforms competitive baselines. 2 related work in the era of statistical machine translation, document-level machine translation has become one of the research focuses in the community of machine translation. @xcite @xcite . recently, with the rapid development of nmt, documentlevel nmt has also gradually attracted people's attention @xcite @xcite @xcite . typically, existing studies aim to improve document-level translation quality with the help of document context, which is usually extracted from neighboring sentences of the current sentence. for example,some researchers applied cache-based models to selectively remember the most relevant context information of the document @xcite @xcite , while some researchers employed hierarchical context networks to catch document context information for transformer @xcite @xcite . specifically, @xcite al. (2018) although these methods have made some progress in document-level nmt, they all ignored the discourse structure information, which can be used to not only enrich word embedding but also guide the selection of relevant context for the current sentence. ## our encoder we propose a novel document-level nmt model based on han @xcite . the difference between ours and han lies in that we introduce the rst-based discourse structure to represent the document-level context, which is incorporated into han to refine translation. figure 1 gives the architecture of our proposed encoder. in addition to the standard encoder for the current sentence, it contains han @xcite as context encoder, and a novel path encoder for the discourse structure. we first use the transformer-based path encoder to model discourse structure information. then, we combine the embedding of each input word with its corresponding path embedding vector and feed the combined vector into the sentence encoder. finally, we use the hierarchical attention network to capture the global contextual embedding and update the hidden states of current sentence as the final output of the whole encoder. in our model, the translation of a document is made by translating each of its sentences sequentially. we introduce discourse structure for both the current sentence and contextual sentences. given a source document x, the translation probability of figure 2 : an example discourse tree of with six edus. n and s denote the relative importance label nucleus and satellite, respectively. sentence 3 is the current sentence to be translated, with two previous context sentences 1 and 2. on the tree, the path marked with dotted lines from the root node to the leaf node e 5 is used to represent the discourse structure of e 5 . ## conclusion this paper has presented a novel discourse structure-based encoder for document-level nmt. the main idea of our encoder lies in enriching the input word embeddings with their path embeddings based on discourse structure. experimental results on english-to-german translation verify the effectiveness of our proposed encoder. in the future, we plan to extend our encoder to other nlp tasks, such as simultaneous translation. simultaneous translation, as well as documentlevel nmt, has difficulty in modeling the long context so that it may be effective to improve the retranslation with the structure information modeled by our encoder. finally, we will focus on refining document-level nmt with variational neural networks, which has shown effecitive in previous studies of sentence-level nmt @xcite .