Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020)

Abstract: Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input. However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns. In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require any external knowledge. Our experiments with different data sizes and multiple language pairs show that fixing the attention heads on the encoder side of the Transformer at training time does not impact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios.

## introduction models based on the transformer architecture @xcite have led to tremendous performance increases in a wide range of downstream tasks @xcite , including machine translation (mt) @xcite . one main component of the architecture is the multi-headed attention mechanism that allows the model to capture longrange contextual information. despite these successes, the impact of the suggested parametrization choices, in particular the self-attention mechanism with its large number of attention heads distributed over several layers, has been the subject of many studies, following roughly four lines of research. the first line of research focuses on the interpretation of the network, in particular on the analysis of attention mechanisms and the interpretability of the weights and connections @xcite @xcite @xcite . a closely related research area attempts to guide the attention mechanism, e.g. by incorporating alignment objectives @xcite , or improving the representation through external information such as syntactic supervision @xcite @xcite . the third line of research argues that transformer networks are over-parametrized and learn redundant information that can be pruned in various ways @xcite . for example, @xcite show that a few attention heads do the "heavy lifting" whereas others contribute very little or nothing at all. similarly, @xcite raise the question whether 16 attention heads are really necessary to obtain competitive performance. finally, several recent works address the computational challenge of modeling very long sequences and modify the transformer architecture with attention operations that reduce time complexity @xcite @xcite @xcite @xcite . this study falls into the third category and is motivated by the observation that most self-attention patterns learned by the transformer architecture merely reflect positional encoding of contextual information @xcite @xcite . from this standpoint, we argue that most attentive connections in the encoder do not need to be learned at all, but can be replaced by simple predefined patterns. to this end, we design, analyze and experimentally compare intuitive and simple fixed attention patterns. the proposed patterns are solely based on positional information and do not require any learnable parameters nor external knowledge. the fixed patterns reflect the importance of locality and pose the question whether encoder self-attention needs to be learned at all to achieve state-of-the-art results in machine translation. this paper provides the following contributions: the results show that fixed self-attention patterns yield consistently competitive results, especially in low-resource scenarios. our results show that the encoder self-attention in transformer-based machine translation can be simplified substantially, reducing the parameter footprint without loss of translation quality, and even improving quality in low-resource scenarios. along with our contributions, we highlight our key findings that give insights for the further development of more lightweight neural networks while retaining state-of-the-art performance for mt: ## related work attention mechanisms in neural machine translation (nmt) were first introduced in combination with recurrent neural networks (rnns) @xcite @xcite , between the encoder and decoder. the transformer architecture extended the mechanism by introducing the so-called self-attention to replace the rnns in the encoder and decoder, and by using multiple attention heads @xcite . this architecture rapidly became the de facto state-of-the-art architecture for nmt, and more recently for language modeling @xcite and other downstream tasks @xcite @xcite . the transformer allows the attention for a token to be spread over the entire input sequence, multiple times, intuitively capturing different properties. this characteristic has led to a line of research focusing on the interpretation of transformer-based networks and their attention mechanisms @xcite @xcite @xcite @xcite @xcite @xcite @xcite . as regards mt, recent work @xcite suggests that only a few attention heads are specialized towards a specific role, e.g., focusing on a syntactic dependency relation or on rare words, and significantly contribute to the translation performance, while all the others are dispensable. at the same time, recent research has attempted to bring the mathematical formulation of selfattention more in line with the linguistic expectation that attention would be most useful within a narrow local scope, e.g. for the translation of phrases @xcite . for instance, @xcite replace the sinusoidal position encoding in the transformer with relative position encoding to improve the focus on local positional patterns. several studies modify the attention formula to bias the attention weights towards local areas @xcite @xcite . @xcite and @xcite use convolutional modules to replace parts of self-attention, making the overall networks computationally more efficient. @xcite mask out certain tokens when computing attention, which favors local attention patterns and prevents redun-dancy in the different attention heads. all these contributions have shown the importance of localness, and the possibility to use lightweight convolutional networks to reduce the number of parameters while yielding competitive results @xcite . in this respect, our work is orthogonal to previous research: we focus only on the original transformer architecture and investigate the replacement of learnable encoder self-attention by fixed, non-learnable attentive patterns. recent analysis papers have identified a certain number of functions to which different selfattention heads tend to specialize. voita et al. (2019b) 2019 ) identify two additional types of heads: bpe-merging heads spread weight over adjacent tokens that are part of the same bpe cluster or hyphenated words, and interrogation heads point to question marks at the end of the sentence. in line with these findings, we design our fixed attention patterns and train nmt models without the need of learning them. in concurrent work, @xcite propose to replace learnable attention weights in transformerbased nmt with hard-coded gaussian distributions. this paper is complementary and differs in several respects: while you et al. (2020) consider three fixed patterns across the encoder-decoder architecture, we focus only on the encoder selfattention but present seven fixed patterns that cover additional known properties of self-attention. we study the relative impact of each of them and analyze their performance with respect to different numbers of encoder-decoder layers, and as semantic feature extractor for lexical ambiguity phenomena. furthermore, in contrast to @xcite , we show that our fixed patterns have a clear beneficial effect in low-resource scenarios. ## methodology in this section, we briefly describe the transformer architecture and its self-attention mechanism, and introduce the fixed attention patterns. ## experiments we perform a series of experiments to evaluate the fixed attentive encoder patterns, starting with a standard german ↔ english translation setup (section 4.1) and then extending the scope to lowresource and high-resource scenarios (section 4.2). we use the opennmt-py @xcite library for training, the base version of transformer as hyper-parameters @xcite , and compare against the reference using sacrebleu @xcite . 3 ## analysis to further analyze the fixed attentive encoder patterns, we perform three targeted evaluations: i) on the sentence length, ii) on the subject-verb agreement task, and iii) on the word sense disambiguation (wsd) task. the length analysis inspects the translation quality by sentence length. the subjectverb agreement task is commonly used to evaluate long-range dependencies @xcite @xcite , while the wsd task addresses lexical ambiguity phenomena, i.e., words of the source language that have multiple translations in the target language representing different meanings @xcite @xcite @xcite . for both tasks, we use contrastive test suites @xcite ) that rely on the ability of nmt systems to score given translations. broadly speaking, a sentence containing the linguistic phenomenon of interest is paired with the correct reference translation and with a modified translation with a specific type of error. a contrast is considered successfully detected if the reference translation obtains a higher score than the artificially modified translation. the evaluation metric corresponds to the accuracy over all decisions. we conduct the analyses using the de-en models from section 4.1, i.e., 8l, 7f token +1l, and 7f token (h8 disabled). ## conclusion in this work, we propose to simplify encoder selfattention of transformer-based nmt models by replacing all but one attention heads with fixed positional attentive patterns that require neither training nor external knowledge. we train nmt models on different data sizes and language directions with the proposed fixed patterns, showing that the encoder self-attention can be simplified drastically, reducing parameter footprint at training time without degradation in translation quality. in low-resource scenarios, translation quality is even improved. our extensive analyses show that i) only adjacent and previous token attentive patterns contribute significantly to the translation performance, ii) the trainable encoder head can also be disabled without hampering translation quality if the number of decoder layers is sufficient, iii) encoder attention heads based on locality patterns are beneficial in low-resource scenarios, but may affect the semantic feature extraction necessary for addressing lexical ambiguity phenomena. apart from the consistent results given by our simple fixed encoder patterns, this work opens up potential further research for simpler and more efficient neural networks for mt, such as synthetic self-attention patterns @xcite .