Contextual Handling in Neural Machine Translation: Look behind, ahead and on both sides (2018)

Abstract: A salient feature of Neural Machine Translation (NMT) is the end-to-end nature of training employed, eschewing the need of separate components to model different linguistic phenomena. Rather, an NMT model learns to translate individual sentences from the labeled data itself. However, traditional NMT methods trained on large parallel corpora with a one-to-one sentence mapping make an implicit assumption of sentence independence. This makes it challenging for current NMT systems to model inter-sentential discourse phenomena. While recent research in this direction mainly leverages a single previous source sentence to model discourse, this paper proposes the incorporation of a context window spanning previous as well as next sentences as source-side context and previously generated output as target-side context, using an effective non-recurrent architecture based on self-attention. Experiments show improvement over non-contextual models as well as contextual methods using only previous context.

## introduction neural machine translation @xcite @xcite has consistently outperformed other mt paradigms across a range of domains, applications and training settings @xcite @xcite the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. and sánchez-cartagena, 2017), thereby emerging as the de facto standard in machine translation. nmt models are typically trained at the sentence level @xcite , whereby the probability of an output sentence given an input sentence is maximized, implicitly making an assumption of sentence independence across the dataset. this works well for the translation of stand-alone sentences or datasets containing shuffled sentences, which are not connected with each other in terms of discursive dependencies. however, in real life situations, written text generally follows a sequential order featuring a number of cross-sentential phenomena. additionally, speech-like texts @xcite exhibit the trait of contextual dependency and sequentiality as well, often containing a greater number of references that require a common knowledge ground and discourse understanding for correct interpretation. figure 1 shows an example of such inter-sentential dependencies. these dependencies are not fully leveraged by the majority of contemporary nmt models, owing to the treatment of sentences as independent units for translation. in order to perform well on sequential texts, nmt models need access to extra information, which could serve as the disambiguating context for better translation. recent work in this direction @xcite @xcite @xcite has primarily focused on previous source-side context for disambiguation. since all of these approaches utilize recurrent architectures, adding context comprising of more than a single previous sentence can be challenging due to either (i) the increased number of estimated parameters and training time, in case of the multi-encoder approach @xcite , or (ii) performance drop due to very long inputs @xcite , in case of extended translation units @xcite . hence, the impact of utilizing a large-sized context window on the source as well as the target side remains unclear. additionally, the impact of incorporating the next sentences as context in the source side also needs to be examined, owing to discourse phenomena like cataphora and gender agreement, illustrated in figure 1 . we address this gap and investigate the contribution of a context window looking behind as well as ahead on the source-side, combined with previous target-side context, in an efficient nonrecurrent "transformer" architecture with selfattention (hereafter transformer), recently proposed by @xcite . we choose this architecture due to its effective handling of longrange dependencies and easily achievable computational parallelization. these characteristics are due to the fact that the transformer is based entirely on self-attention, as opposed to lstms or grus. the non-recurrent architecture enables effective parallelization, which is not possible with rnns due to their sequentiality, thereby reducing the computational complexity considerably. we perform experiments using differently sized context windows on the source and target side. this is the first effort towards contextual nmt with transformer to the best of our knowledge. on the english-italian data from the iwslt 2017 shared task @xcite , the best of our models achieves a 2.3% increase in bleu score over a baseline transformer model trained without any inter-sentential context and a 2.6% increase in bleu score over a multi-source bilstm model trained using a previous source sentence as addi-tional context. the major contributions of this paper are summarized below: the rest of the paper is organized as follows: we describe an outline of the related work in section 2, and provide a theoretical background in section 3. section 4.1 briefly describes the discourse phenomena which we would like to capture using our contextual nmt models. our approach to model discourse and the experiments conducted are described in section 4. section section 5 presents the results obtained by our models, along with a linguistic analysis of the implications therein. we present the conclusions of the present research and highlight possible directions for future work in section 6. ## related work discourse modeling has been explored to a significant extent for statistical machine translation @xcite , using methods like discriminative learning @xcite , context features @xcite @xcite , bilingual language models @xcite , document-wide decoding @xcite and factored models @xcite . the majority of these works, however, look mainly at intra-sentential discourse phenomena, owing to the limited capability of smt models to exploit extra-sentential context. the neural mt paradigm, on the other hand, offers a larger number of avenues for looking beyond the current sentence during translation. recent work on incorporating contextual information into nmt models has delved primarily into multi-encoder models @xcite @xcite , hierarchy of rnns @xcite and extended translation units containing the previous sentence (tiedemann and scherrer, 2017). these approaches build upon the multi-task learning method proposed by @xcite , adapting it specifically for translation. @xcite propose a multi-source training method, which employs multiple encoders to represent inputs coming from different languages. their method utilizes the sources available in two languages in order to produce better translations for a third language. @xcite use the multi-encoder framework, with one set of encoder and attention each for the previous and the current source sentence as an attempt to model context. however, this method would be computationally expensive with an increase in the number of contextual sentences owing to the increase in estimated parameters. @xcite employ a hierarchy of rnns to summarize source-side context (previous three sentences). this method addresses the computational complexity to an extent, however it does not incorporate target-side context, which has been shown to be useful by @xcite . @xcite present an in-depth analysis of the evaluation of discourse phenomena in nmt and the challenges faced thereof. they provide a hand-crafted test set specifically aimed at capturing discursive dependencies. however, this set is created with the assumption that the disambiguating context lies in the previous sentence, which is not always the case @xcite . our work is most similar to @xcite , who employ the standard nmt architecture without multiple encoders, but using larger blocks containing the previous and the current sentence as input for the encoder, as an attempt to better model discourse phenomena. the primary limitation of this method is the inability to add larger context due to the ineffective handling of long-range dependencies by rnns @xcite . additionally, this method does not look at the following source-text, due to which phenomena like cataphora and lexical cohesion are not captured well. while the above-mentioned works employ the previous source text, we propose employing a context window spanning previous as well as next source sentences in order to model maximal discourse phenomena. on the target-side, we decode the previous and current sentence while looking at the source-window, thereby employing targetside context as well. additionally, we employ the transformer for our contextual models, as opposed to the above-mentioned works using rnns, due to the enhanced long-range performance and computational parallelization. ## conclusion neural mt methods, being typically trained at sentence level, fail to completely capture implicit discourse relations established at the inter-sentential level in the text. in this paper, we demonstrated that looking behind as well as peeking ahead in the source text during translation leads to better performance than translating sentences in isolation. additionally, jointly decoding the previous as well as current text on the target-side helps to incorporate target-side context, which also shows improvement in translation quality to a certain extent, albeit being more prone to error propagation with increase in the size of the context window. moreover we showed that using the transformer architecture, a simple strategy like concatenation of the context yields better performance on spoken texts than non-contextual models, whilst being trained significantly faster than recurrent architectures. contextual handling using self-attention is hence a promising direction to explore in the future, possibly with multi-source techniques in conjugation with the transformer architecture. in the future, we would like to perform a fine-grained analysis on the improvement observed for specific linguistic phenomena using our extended context models.