Multilayer Representation Fusion for Neural Machine Translation Neural machine translation systems require a number of stacked layers for deep models. But the prediction depends on the sentence representation of the topmost layer with no access to lowlevel representations. This makes it more difficult to train the model and poses a risk of information loss to prediction. In this paper, we propose a multilayer representation fusion (MLRF) approach to fusing stacked layers. In particular, we design three fusion functions to learn a better representation from the stack. Experimental results show that our approach yields improvements of 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT GermanEnglish and NIST ChineseEnglish MT tasks respectively. The result is new stateoftheart in GermanEnglish translation.

Selective Attention for Contextaware Neural Machine Translation Despite the progress made in sentencelevel NMT, current systems still fall short at achieving fluent, good quality translation for a full document. Recent works in contextaware NMT consider only a few previous sentences as context and may not scale to entire documents. To this end, we propose a novel and scalable topdown approach to hierarchical attention for contextaware NMT which uses sparse attention to selectively focus on relevant sentences in the document context and then attends to key words in those sentences. We also propose singlelevel attention approaches based on sentence or wordlevel information in the context. The documentlevel context representation, produced from these attention modules, is integrated into the encoder or decoder of the Transformer model depending on whether we use monolingual or bilingual context. Our experiments and evaluation on EnglishGerman datasets in different document MT settings show that our selective attention approach not only significantly outperforms contextagnostic baselines but also surpasses contextaware baselines in most cases.