Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation (2020)

Abstract: In encoder-decoder neural models, multiple encoders are in general used to represent the contextual information in addition to the individual sentence. In this paper, we investigate multi-encoder approaches in document-level neural machine translation (NMT). Surprisingly, we find that the context encoder does not only encode the surrounding sentences but also behaves as a noise generator. This makes us rethink the real benefits of multi-encoder in context-aware translation - some of the improvements come from robust training. We compare several methods that introduce noise and/or well-tuned dropout setup into the training of these encoders. Experimental results show that noisy training plays an important role in multi-encoder-based NMT, especially when the training data is small. Also, we establish a new state-of-the-art on IWSLT Fr-En task by careful use of noise generation and dropout methods.

## introduction sentence-level neural machine translation (nmt) systems ignore the discourse phenomena and encode the individual source sentences with no use of contexts. in recent years, the context-aware models which learn contextual information from surrounding sentences have shown promising results in generating consistent and coherent translations @xcite @xcite @xcite @xcite . there are two common approaches to incorporating contexts into nmt: the simple way is to concatenate the context and the current sentence to form a context-aware input sequence @xcite , whereas a more widely-used approach utilizes additional neural networks to encode context sentences @xcite @xcite . here we name the former as the single-encoder approach and name the latter as the multi-encoder approach. however, large-scale document corpora are not easily available. most context-aware nmt systems are evaluated on small datasets and significant bleu improvements are reported @xcite @xcite . in our experiments, we find that the improvement persists if we feed pseudo sentences into the context encoder, especially when we train the system on small-scale data. a natural question here is: how much does the improvement come from the leverage of contextual information in multi-encoder? in this work, we aim to investigate what kinds of information that the context-aware model captures. we re-implement several widely used context-aware architectures based on the multiencoder paradigm, and do an in-depth analysis to study whether the context encoder captures the contextual information. by conducting extensive experiments on several document-level translation benchmarks, we observe that: encoderc encoders attention attention context source target hc hs h c decoder ⊕ ... ... our contribution is two folds: (i) we find that the benefit of the multi-encoder context-aware approach is not from the leverage of contextual information. instead, the context encoder acts more like a noise generator to provide richer training signals. (ii) the finding here inspires us to develop a simple yet effective training strategy: we add a gaussian-noise to the encoder output, which can effectively alleviate the overfitting, especially on small datasets. ## approaches to incorporating contexts into nmt here we describe two ways of introducing contextual information into nmt systems. ## results and discussion to study whether the context-encoder network captures contextual information in training, we present three types of context as the input of the contextencoder: ## related work context-aware nmt systems incorporating the contextual information generate more consistent and coherent translations than sentence-level n-mt systems. most of the current context-aware nmt models can be classified into two main categories, single-encoder systems @xcite and multi-encoder systems @xcite @xcite . voita et al. (2018) and zhang et al. (2018) integrated an additional encoder to leverage the contextual information into transformer-based nmt systems. miculicich et al. (2018) employed a hierarchical attention network to model the contextual information. maruf and haffari (2018) built a context-aware nmt system using a memory network, and maruf et al. (2019) encoded the whole document with selective attention network. however, most of the work mentioned above utilized more complex modules to capture the contextual information, which can be approximately regarded as multi-encoder systems. for a fair evaluation of context-aware nmt methods, we argue that one should build a strong enough sentence-level baseline with carefully regularized methods, especially on small datasets @xcite . beyond this, @xcite and @xcite acknowledged that bleu score is insufficient to evaluate context-aware models, and they emphasized that multi-encoder architectures alone had a limited capacity to exploit discourse-level context. in this work, we take a further step to explore the main cause, showing that the context-encoder acts more like a noise generator, and the bleu improve-ments mainly come from the robust training instead of the leverage of contextual information. additionally, @xcite added the gaussian noise to word embedding to simulate lexical-level perturbations for more robust training. differently, we added the gaussian noise to the encoder output which plays a similar role with context-encoder, which provides additional training signals. ## conclusions we have shown that, in multi-encoder contextaware nmt, the bleu improvement is not attributed to the leverage of contextual information. even though we feed the incorrect context into training, the nmt system can still obtain substantial bleu improvements on several small datasets. another observation is that the nmt models can even achieve better translation quality without the context encoder. this gives us an interesting finding that the context-encoder acts more like a noise generator, which provides rich supervised training signals for robust training. motivated by this, we significantly improve the sentence-level systems with a gaussian noise imposed on the encoder output. experiments on large-scale training data demonstrate the effectiveness of this method.