Exploring Discourse Structure in Document-level Machine Translation (2023)

Abstract: Neural machine translation has achieved great success in the past few years with the help of transformer architectures and large-scale bilingual corpora. However, when the source text gradually grows into an entire document, the performance of current methods for document-level machine translation (DocMT) is less satisfactory. Although the context is beneficial to the translation in general, it is difficult for traditional methods to utilize such long-range information. Previous studies on DocMT have concentrated on extra contents such as multiple surrounding sentences and input instances divided by a fixed length. We suppose that they ignore the structure inside the source text, which leads to under-utilization of the context. In this paper, we present a more sound paragraph-to-paragraph translation mode and explore whether discourse structure can improve DocMT. We introduce several methods from different perspectives, among which our RST-Att model with a multi-granularity attention mechanism based on the RST parsing tree works best. The experiments show that our method indeed utilizes discourse information and performs better than previous work.

## introduction transformer @xcite based approaches, together with adequate bilingual datasets, have led to significant progress on machine translation (mt). however, the performance of mt models usually drops dramatically when processing long texts. although document-level machine translation (docmt) can be solved with sentencelevel mt by translating each sentence separately, the potential information in the long-range context may be ignored. to address these problems, many methods in docmt have been proposed to better utilize the contextual information and improve the overall translation quality of the document. among these methods, the dominant approaches still adhere to the sentence-by-sentence mode, but they utilize additional contextual information, including the surrounding sentences @xcite @xcite @xcite , document contextual representation @xcite and memory units @xcite . in recent years, many researches have turned to translating multiple sentences or the entire document at once @xcite @xcite . however, previous work @xcite @xcite has demonstrated that direct doc2doc translation may cause the model not to converge. therefore, those methods adopt pertinent measures such as data augmentation, text truncation, and specific frameworks for sentence alignment. despite the effectiveness of previous efforts on docmt, some research, such as @xcite , suggested that existing methods may not fully utilize the context and the improvement may come from regularization. @xcite also indicated that dynamic context selected from the surrounding sentences can improve translation more efficiently. the additional contexts used in most previous work are simply based on the distance from the current sentence or the length of the text, which is somewhat arbitrary. in this paper, we aim to explore a more reasonable way to encode context through the discourse structure. discourse structure refers to how elementary text units are organized to form a discourse and logically linked to one another. early studies have demonstrated that discourse parsing can benefit various downstream nlp tasks, including sentiment analysis @xcite , relation extraction @xcite ), text summarization @xcite , machine translation evaluation @xcite @xcite and so on. rst parsing, based on rhetorical structure theory @xcite , is one of the most influential parsing methods in discourse analysis. according to rst, a text is segmented into several clause-like units (edus) as leaves of the corresponding parsing tree. through certain rhetorical relations among adjacent spans, underlying edus or larger text spans are recursively linked and merged to form their parent nodes, representing the concatenation of them. although rst parsing would be better conducted on an integral text to maintain discourse structure, existing models perform poorly on long texts. as a result, we present a new paragraph-toparagraph translation mode, where the original document is divided into several shorter paragraphs. our paragraph segmentation is generated by the texttiling tool @xcite based on subtopic shifts and discourse cues, since frequently-used datasets of docmt do not contain paragraph alignment tags. we suppose the discourse analysis of paragraphs is a proper compromise and more sound than previous partitioning methods. and it is more labor-saving when dealing with other multilingual corpora equipped with paragraph alignments. we employ the end-to-end method of @xcite to train our rst parsing model. the parsing task is reformulated into a seq2seq task through a linearization process and then trained based on a pretrained language model. several attempts have been made from different perspectives to investigate how to utilize the discourse structure, including the integration of the rst sequence, an rst-specific attention mechanism, and graph learning based on the rst tree. among them, our rst-att model, designed with a multi-granularity attention mechanism to inject the rst tree into the translation encoder, achieves the best performance. specifically, each token at the first encoder layer can only notice other tokens from the edu it belongs to, when calculating self-attention scores. as the encoder layer moves backward, the range of attention continuously expands to complete context according to the structure of the rst tree. we believe such a progressive pattern can reduce the difficulty of modeling long-range context. overall, our main contributions are as follows: ## methodology in this section, we will elaborate on the relevant steps to obtain the discourse structure information and explore its utilization in docmt. in section 2.1, we compare two text segmentation methods and determine the texttiling tool for our paragraph segmentation according to the distribution of results. in section 2.2, we introduce the brief training process of the rst parsing model and how to linearize the corresponding rst tree. section 2.3 describes the construction and details of our proposed rst sequence integration method, as well as the rst-att model with the rst-specific attention mechanism of multiple levels of granularity. ## datasets and settings we evaluate our models on three widely used datasets for document-level machine translation for english to german, from @xcite . ted the corpus includes ted talks and corresponding translations from iwslt 2017. tst2016-2017 is used as the test set and the rest as valid. news the corpus comes from news commentary v11. newstest2016 is used as the test set and newstest2015 as valid. ## experiments and analysis we compare the experimental results of our methods with previous approaches to document-level machine translation on ted, news, and europarl, as shown in and we categorize all methods based on whether they are based on pretrained language models. above all, we introduce our para2para baseline, which is finetuned on the original mbart25 model. it simply performs paragraph-to-paragraph translation based on the segmented paragraphs without utilizing any rst discourse information. for better comparison, we also present the results of the traditional sub-document baseline sdoc2sdoc from @xcite . as described in section 1, many studies on docmt have adopted splitting each document into sub-documents for translation based on a specified maximum length, since standard doc2doc translation was pointed out to be prone to training failures. the results show that our para2para baseline outperforms the traditional sdoc2sdoc with the same mbart25 model, proving the improvement of our new translation mode. it can be attributed to the fact that simple sdoc2sdoc mode, without any other optimization, may arbitrarily ignore the interactions between strongly related sentences. in contrast, our proposed paragraph-to-paragraph method takes them into consideration and is expected to make better use of contextual information. furthermore, we introduce discourse information and propose our rst-seq and rst-att models. both of them follow the para2para translation mode based on mbart25, and contain the same number of parameters as the baselines mentioned above. although rst-seq performs better than the para2para baseline, the improvements are not prominent. we suppose that much longer inputs of linearized rst trees may increase the difficulty of training. moreover, it may be difficult for mbart25 to directly adapt to the format of the rst sequence since it in addition, our rst-att model achieves the best results and improves d-bleu scores by 1.07, 0.57, and 0.06 on ted, news, and europarl, respectively, showing the effectiveness of our proposed attention mechanism. its superiority over the rst-seq demonstrates that discourse knowledge may be applied to the model structure more effectively than the input data. furthermore, ## conclusions in this paper, we explore the role of discourse structure in document-level machine translation. we introduce a more sound paragraph-to-paragraph translation mode than the several surrounding sentences or fixed length of texts used in previous studies. to better take advantage of the rst parsing tree, we propose the rst-att model with a multigranularity attention mechanism depending on the tree structure. the experiment results prove the superiority of our method, and further evaluation indicates that both the guidance of discourse structure and more levels of granularity contribute to the improvement. and the more effective utilization of rst labels for docmt is left to future research.