Selective Attention for Contextaware Neural Machine Translation Despite the progress made in sentencelevel NMT, current systems still fall short at achieving fluent, good quality translation for a full document. Recent works in contextaware NMT consider only a few previous sentences as context and may not scale to entire documents. To this end, we propose a novel and scalable topdown approach to hierarchical attention for contextaware NMT which uses sparse attention to selectively focus on relevant sentences in the document context and then attends to key words in those sentences. We also propose singlelevel attention approaches based on sentence or wordlevel information in the context. The documentlevel context representation, produced from these attention modules, is integrated into the encoder or decoder of the Transformer model depending on whether we use monolingual or bilingual context. Our experiments and evaluation on EnglishGerman datasets in different document MT settings show that our selective attention approach not only significantly outperforms contextagnostic baselines but also surpasses contextaware baselines in most cases.

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed The utility of linguistic annotation in neural machine translation seemed to had been established in past papers. The experiments were however limited to recurrent sequencetosequence architectures and relatively small data settings. We focus on the stateoftheart Transformer model and use comparably larger corpora. Specifically, we try to promote the knowledge of sourceside syntax using multitask learning either through simple data manipulation techniques or through a dedicated model component. In particular, we train one of Transformer attention heads to produce sourceside dependency tree. Overall, our results cast some doubt on the utility of multitask setups with linguistic information. The data manipulation techniques, recommended in previous works, prove xc2xa0ineffective in large data settings. The treatment of selfattention as dependencies seems much more promising: it helps in translation and reveals that Transformer model can very easily grasp the syntactic structure. An important but curious result is, however, that identical gains are obtained by using trivial linear treesu0027u0027 instead of true dependencies. The reason for the gain thus may not be coming from the added linguistic knowledge but from some simpler regularizing effect we induced on selfattention matrices.