Multi-Hop Transformer for Document-Level Machine Translation (2021)

Abstract: Document-level neural machine translation (NMT) has proven to be of profound value for its effectiveness on capturing contextual information. Nevertheless, existing approaches 1) simply introduce the representations of context sentences without explicitly characterizing the inter-sentence reasoning process; and 2) feed ground-truth target contexts as extra inputs at the training time, thus facing the problem of exposure bias. We approach these problems with an inspiration from human behavior – human translators ordinarily emerge a translation draft in their mind and progressively revise it according to the reasoning in discourse. To this end, we propose a novel Multi-Hop Transformer (MHT) which offers NMT abilities to explicitly model the human-like draft-editing and reasoning process. Specifically, our model serves the sentence-level translation as a draft and properly refines its representations by attending to multiple antecedent sentences iteratively. Experiments on four widely used document translation tasks demonstrate that our method can significantly improve document-level translation performance and can tackle discourse phenomena, such as coreference error and the problem of polysemy.

## introduction neural machine translation (nmt) employs an endto-end framework @xcite and has advanced promising results on various sentencelevel translation tasks @xcite @xcite . however, most of nmt models handle sentences independently, regardless of the linguistic context that may appear outside the current sentence @xcite . this makes nmt insufficient to fully resolve the typical context-dependent phenomena problematic, † these authors contributed equally to this work. * corresponding author. e.g. coreference @xcite , lexical cohesion @xcite , as well as lexical disambiguation @xcite . recent studies @xcite @xcite @xcite @xcite @xcite have proven to be effective on tackling discourse phenomena via feeding nmt with contextual information, e.g. sourceside @xcite @xcite or target-side context sentences @xcite . despite their successes, these methods simply merge the representations of context sentences together, lacking a mechanism to explicitly characterize the inter-sentence reasoning upon the context. another shortage in existing document-level nmt is the problem of exposure bias. most of methods utilized the ground-truth target context for training but the generated translations for inference, leading to inconsistent inputs at training and testing time @xcite . intuitively, human translators tend to acquire useful context information from the reasoning process among sentences, thus figuring out the correct meaning when they encounter ambiguity during translation. @xcite and @xcite empirically verified that modeling multi-hop reasoning among sentences benefits to the language understanding task, e.g text comprehension. @xcite showed that documentlevel nmt model can profit from relative positions with respect to context sentences, which to some extent confirms the importance of the relationship among sentences. meanwhile, @xcite demonstrated that sentence-level nmt could be improved by a two-pass draft-editing process, of which the second-pass decoder refines the target sentence generated by a first-pass standard decoder. accordingly, we propose to improve document-level nmt using a novel framework -multi-hop transformer, which imitates draft-editing and reasoning process of human translators. specifically, we implement an explicit reasoning process by exploiting source and target antecedent sentences with concurrently stacked attention layers, thus performing the progressive refinement on the representations of the current sentence and its translation. besides, we leverage the draft to present context information on the target side during both training and testing, alleviating the problem of exposure bias. we conduct experiments on four widely used document translation tasks: english-german and chinese-english ted, english-russian opensubtitles, as well as english-german europarl-7 datasets. experimental results demonstrate that our method significantly outperforms both context-agnostic and context-aware methods. the qualitative analysis confirms the effectiveness of the proposed multihop reasoning mechanism on resolving many linguistic phenomena, such as word sense disambiguation and coreference resolution. our contributions are mainly in: ## preliminary transformer nmt is an end-to-end framework to build translation models. @xcite propose a new architecture called transformer which adopts self-attention network for both encoding and decoding. both its encoder and decoder consist of multiple layers, each of which includes a multi-head self-attention and a feed-forward sublayer. additionally, each layer of the decoder applys a multi-head cross attention to capture information from the encoder. transformer has shown superiority in a variety of nlp tasks. therefore, we construct our models upon this advanced architecture. document-level nmt in order to correctly translate the sentence with discourse phenomena, nmt models need to look beyond the current sentence and integrate contextual sentences as auxiliary inputs. formally, let @xmath0, x 2 , ..., x i ) be a source-language document composed of i sentences, where related work several studies have explored multi-input models to leverage the contextual information from source-side @xcite or target-side sentences @xcite . for the former, @xcite propose a new encoder to represent document-level context from previous source-side sentences . @xcite and junczys-dowmunt (2019) utilize the concatenation of previous source-side sentences as input, while @xcite make use of gate mechanism to balance the weight between current source sentence and its context. for the latter, @xcite propose a hierarchical attention (han) framework to capture the target contextual information in the decoder. @xcite , @xcite and @xcite take both source-side and target-side context into account. motivation as seen, both of the existing methods simply introduce the context sentences without explicitly characterizing the inter-sentence reasoning. intuitively, when humans have difficulty in translation like encountering ambiguity phenomenon, they could acquire more information source-side context encoder source embedding k-th hop attention layer second hop attention layer first hop attention layer self-attention layer source-side context encoder source-side context encoder target embedding k-th hop attention layer second hop attention layer first hop attention layer draft-attention layer 1 -encoder-decoder attention layer + 1 -+ target-side context encoder self-attention layer target-side draft encoder softmax � � � -� � � � -�+� � � � -� � � � � -� � � � -�+� � � � -� 6 × 6 × translation source sentence i target sentence i draft sentence context draft sentence i-1 context draft sentence i-k+1 context draft sentence i-k source context sentence i-1 source context sentence i-k+1 source context sentence i-k source-side sentence encoder multi-hop encoder multi-hop decoder target-side sentence encoder target-side context encoder target-side context encoder … … … … … … figure 1: illustration of multi-hop transformer. c i-j s and c i-j t indicate the j th previous sentence in the source side and target side respectively. d denotes the draft of current source sentence. all drafts are generated by a pre-trained sentence-level nmt model. the modules inside dashed box are the proposed multi-hop attention layers, which gradually refine the representation of current sentence. finally, the context gate α is used to control the contextual information. from the contexts sentence by sentence and then perform reasoning to figure out the exact meaning. we attribute that such reasoning process is also beneficial to machine translation task. recent successes in text comprehension communities have to some extent supported our hypothesis @xcite . for example, @xcite propose a multi-hop end-to-end memory network, which can renew the query representation with multiple computational steps (which they term "hops"). @xcite extend an attention-sum reader to multi-turn reasoning with a gating mechanism. in addition, @xcite introduce multi-hop attention, which used multiple turns to effectively exploit and reason over the relation among queries and documents. in this paper, we propose to bring the idea of multi-hop into document translation and aim at mimicking the multi-step comprehension and revising process of human translators. contrast with those models for text comprehension which scan the query and document for multiple passes, our model iteratively focuses on different context sentences, which captures the inter-sentence reasoning semantics of contextual sentences to incrementally refine the representation of current sentence. ## multi-hop transformer with this mind, we propose a novel method called multi-hop transformer, which models the reasoning process among multiple contextual sentences in both source side and target side. the source-side contexts are directly acquired from the document. the target-side contexts, called target-side drafts in this paper, are generated by a sentence-level nmt model. these contexts are fed into the multi-hop transformer with pre-trained encoders. the overall architecture of our proposed model is illustrated in figure 1 , which consists of three components: ## analysis in this section, to gain further insight, we explore the effectiveness of several factors of our model, including 1) multi-hop attention; 2) context window size; 3) reasoning direction; 4) sides for introducing context; and 5) target contexts. moreover, we show qualitative analysis on discourse phenomena to better understand the advantage of our model. ## conclusion in this paper, we propose a novel document-level translation model called multi-hop transformer case 1 case 2 context3 they were 12, 3, and 1 when the hum stopped. so if your show gets a rating of nine points ... context2 the hum of the engine died. then you have a top two percent show. context1 i stopped loving work. i couldn't restart the engine. that's shows like "breaking bad," ... source the hum would not come back. that kind of show . reference das summen kam nicht zurück. diese art von show . transformer der summen kam nicht zurück . das zeigt . ca-transformer der summen kam nicht zurück . das zeigt . ca-han das summen würde nicht zurückkommen. das zeigt irgendwie . mht das summen kam nicht zurück . diese art von show . with an inspiration from human reasoning behavior to explicitly model the human-like draft-editing and reasoning process. experimental results on four widely used tasks show that our model can achieve better performance than both context-agnostic and context-aware strong baseline. furthermore, the qualitative analysis shows that the multi-hop reasoning mechanism is capable of solving some discourse phenomena by capturing context semantics more accurately.