Document Graph for Neural Machine Translation (2021)

Abstract: Previous works have shown that contextual information can improve the performance of neural machine translation (NMT). However, most existing document-level NMT methods failed to leverage contexts beyond a few set of previous sentences. How to make use of the whole document as global contexts is still a challenge. To address this issue, we hypothesize that a document can be represented as a graph that connects relevant contexts regardless of their distances. We employ several types of relations, including adjacency, syntactic dependency, lexical consistency, and coreference, to construct the document graph. Then, we incorporate both source and target graphs into the conventional Transformer architecture with graph convolutional networks. Experiments on various NMT benchmarks, including IWSLT English–French, Chinese-English, WMT English–German and Opensubtitle English–Russian, demonstrate that using document graphs can significantly improve the translation quality. Extensive analysis verifies that the document graph is beneficial for capturing discourse phenomena.

## introduction although neural machine translation (nmt) has achieved great success on sentence-level translation tasks, many studies pointed out that translation mistakes become more noticeable at the documentlevel @xcite @xcite @xcite @xcite @xcite @xcite . they proved that these mistakes can be alleviated by feeding the contexts into context-agnostic nmt models. previous works have explored various methods to integrate context information into nmt models. they usually take a limited number of previous sentences as contexts and learn context-aware representations using hierarchical networks @xcite @xcite or extra context encoders @xcite @xcite . different from representation-based approaches, @xcite and @xcite propose using a cache to memorize context information, which can be either history hidden states or lexicons. to keep tracking of most recent contexts, the cache is updated when new translations are generated. therefore, long-distance contexts would likely be erased. how to use long-distance contexts is drawing attention in recent years. approaches, like treating the whole document as a long sentence (junczys-dowmunt, 2019) and using memory and hierarchical structures @xcite @xcite , are proposed to take global contexts into consideration. however, @xcite point out that not all the words in a document are beneficial to context integration, suggesting that it is essential for each word to focus on its own relevant context. to address this problem, we suppose to build a document graph for a document, where each word is connected to those words which have a direct influence on its translation. figure 1 shows an example of a document graph. explicitly, a document graph is defined as a directed graph where: (1) each node represents a word in the document; (2) each edge represents one of the following relations between words: (a) adjacency; (b) syntactic dependency; (c) lexical consistency; or (d) coreference. we apply a graph convolutional network (gcn) on the document graph to obtain a document-level contextual representation for each word, fed to the conventional transformer model @xcite by additional attention and gating mechanisms. we evaluate our model on four translation benchmarks, iwslt english-french (en-fr) and chinese-english (zh-en), opensubtitle english-russian (en-ru), and wmt english-german (en-de). experimental results demonstrate that our approach is consistently superior to previous works @xcite @xcite @xcite on all the language pairs. contributions of this work are summarized as: ## approach in this section, we introduce the proposed document graph and model for leveraging contextual information from documents. firstly, we present a definition of the problem. then, the construction and representation learning of document graphs are explained in section 2.2 and section 2.3, respectively. finally, we describe the method of integrat-corenlp ( https://corenlp.run/ ). ## analysis in this section, we analyze the proposed method to reveal its strengths and weaknesses in terms of (1) ablation bleu para. sentencen embedding √ 31.66 20.21k × 31.46 20.00k ## conclusion in this paper, we propose a graph-based approach for document-level translation, which leverages both source and target contexts. graphs are constructed according to inter-sentential and intrasentential relations. we employ a gcn-based graph encoder to learn the graph representations, which are then fed into the nmt model via attention and gating mechanisms. experiments on four translation tasks and several existing architectures show the proposed approach consistently improves translation quality across different language pairs. further analyses demonstrate the effectiveness of graphs and the capability of leveraging long-distance context. in the future, we would like to enrich the types of relations to cover more document phenomena.