Enhancing Machine Translation with Dependency-Aware Self-Attention (2020)

Abstract: Most neural machine translation models only rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. In this work, we investigate different approaches to incorporate syntactic knowledge in the Transformer model and also propose a novel, parameter-free, dependency-aware self-attention mechanism that improves its translation quality, especially for long sentences and in low-resource scenarios. We show the efficacy of each approach on WMT English-German and English-Turkish, and WAT English-Japanese translation tasks.

## introduction research in neural machine translation (nmt) has mostly exploited corpora consisting of pairs of parallel sentences, with the assumption that a model can automatically learn prior linguistic knowledge via an attention mechanism @xcite . however, @xcite found that these models still fail to capture deep structural details, and several studies @xcite @xcite have shown that syntactic information has the potential to improve these models. nevertheless, the majority of syntax-aware nmt models are based on recurrent neural networks (rnns; @xcite , with only a few recent studies that have investigated methods for the transformer model @xcite . @xcite evaluated an approach to incorporate syntax in nmt with a transformer model, which not only required three encoders and two decoders, but also target-side dependency relations (precluding its use to low-resource target languages). @xcite integrate source-side syntax by concatenating the intermediate representations of a dependency parser to word embeddings. in contrast to ours, this approach does not allow to learn sub-word units at the source side, requiring a larger vocabulary to minimize out-of-vocabulary words. @xcite interleave words with syntax representations which results in longer sequences -requiring gradient accumulation for effective training -while only leading to +0.5 bleu on wat ja-en when using ensembles of transformers. finally, @xcite propose two simple data augmentation techniques to incorporate source-side syntax: one that works well on low-resource data, and one that achieves a high score on a large-scale task. our approach, on the other hand, performs equally well in both settings. while these studies improve the translation quality of the transformer, they do not exploit its properties. in response, we propose to explicitly enhance the its self-attention mechanism (a core component of this architecture) to include syntactic information without compromising its flexibility. recent studies have, in fact, shown that self-attention networks benefit from modeling local contexts by reducing the dispersion of the attention distribution @xcite @xcite , and that they might not capture the inherent syntactic structure of languages as well as recurrent models, especially in low-resource settings @xcite . here, we present parentscaled self-attention (pascal): a novel, parameterfree local attention mechanism that lets the model focus on the dependency parent of each token when encoding the source sentence. our method is simple yet effective, improving translation quality with no additional parameter or computational overhead. our main contributions are: ## model in order to design a neural network that is efficient to train and that exploits syntactic information while producing high-quality translations, we base our model on the transformer architecture @xcite and upgrade its encoder with parent-scaled self-attention (pascal) heads at layer l s . pascal heads enforce contextualization from the syntactic dependencies of each source token, and, in practice, we replace standard selfattention heads with pascal ones in the first layer as its inputs are word embeddings that lack any contextual information. our pascal sub-layer has the same number h of attention heads as other layers. source syntax similar to previous work, instead of just providing sequences of tokens, we supply the encoder with dependency relations given by an external parser. our approach explicitly exploits sub-word units, which enable open-vocabulary translation: after generating sub-word units, we compute the middle position of each word in terms of number of tokens. for instance, if a word in position 4 is split into three tokens, now in positions 6, 7 and 8, its middle position is 7. we then map each sub-word of a given word to the middle position of its parent. for the root word, we define its parent to be itself, resulting in a parse that is a directed graph. the input to our encoder is a sequence of t tokens and the absolute positions of their parents. ## conclusion this study provides a thorough investigation of approaches to induce syntactic knowledge into self-attention networks. through extensive evaluations on various translation tasks, we find that approaches effective for rnns do not necessarily transfer to transformers (e.g. +s&h). conversely, dependency-aware self-attention mechanisms (lisa and pascal) best embed syntax, for all corpus sizes, with pascal consistently outperforming other all approaches. our results show that exploiting core components of the transformer to embed linguistic knowledge leads to higher and consistent gains than previous approaches.