Hard-Coded G aussian Attention for Neural Machine Translation (2020)

Abstract: Recent work has questioned the importance of the Transformer’s multi-headed attention for achieving high translation quality. We push further in this direction by developing a “hard-coded” attention variant without any learned parameters. Surprisingly, replacing all learned self-attention heads in the encoder and decoder with fixed, input-agnostic Gaussian distributions minimally impacts BLEU scores across four different language pairs. However, additionally, hard-coding cross attention (which connects the decoder to the encoder) significantly lowers BLEU, suggesting that it is more important than self-attention. Much of this BLEU drop can be recovered by adding just a single learned cross attention head to an otherwise hard-coded Transformer. Taken as a whole, our results offer insight into which components of the Transformer are actually important, which we hope will guide future work into the development of simpler and more efficient attention-based models.

## introduction the transformer (vaswani et al., 2017) has become the architecture of choice for neural machine translation. instead of using recurrence to contextualize source and target token representations, transformers rely on multi-headed attention mechanisms (mha), which speed up training by enabling parallelization across timesteps. recent work has called into question how much mha contributes to translation quality: for example, a significant fraction of attention heads in a pretrained transformer can be pruned without appreciable loss in bleu (voita et al., 2019; michel et al., 2019), and self-attention can be replaced by less expensive modules such as convolutions (yang et al., 2018; wu et al., 2019). in this paper, we take this direction to an extreme by developing a variant of mha without * authors contributed equally. ## background in this section, we first briefly review the transformer architecture of vaswani et al. (2017) with a focus on its multi-headed attention. then, we provide an analysis of the learned attention head distributions of a trained transformer model, which motivates the ideas discussed afterwards. ## hard-coded gaussian attention while learned attention enables model flexibility (e.g., a head can "look" far away from the current position if it needs to), it is unclear from the above analysis how crucial this flexibility is. to examine this question, we replace the attention distribution computation in equation 1 (i.e., scaled dot product of queries and keys) with a fixed gaussian distribution. @xcite in doing so, we remove all learned parameters from the attention computation: the mean of the gaussian is determined by the position i of the current query token, and the standard deviation is always set to 1. @xcite as transformers contain both self-attention and cross attention, the rest of this section details how we replace both of these components with simplified versions. we will re-fer to experimental results on the relatively small iwslt16 english-german dataset throughout this section to contextualize the impact of the various design decisions we describe. section 4 contains a more fleshed out experimental section with many more datasets and language pairs. ## large-scale experiments the previous section developed hard-coded configurations and presented results on the relatively small iwslt16 en-de dataset. here, we expand our experiments to include a variety of different datasets, language pairs, and model sizes. for all hard-coded head configurations, we use the optimal iwslt16 en-de setting detailed in section 3.1 and perform no additional tuning on the other datasets. this configuration nevertheless proves robust, as we observe similar trends with our hard-coded transformers across all of datasets. foot_7 ## bigger batches & decoding speedups we have thus far motivated our work as an exploration of which components of the transformer are necessary to obtain high translation quality. our results demonstrate that encoder and decoder self-attention can be replaced with hard-coded attention distributions without loss in bleu, and that mha brings minor improvements over singleheaded cross attention. in this section, we measure efficiency improvements in terms of batch size increases and decoding speedup. ## analysis taken as a whole, our experimental results suggest that many of the components in the transformer can be replaced by highly-simplified versions without adversely affecting translation quality. in this section, we explain how hard-coded self-attention does not degrade translation quality (section 6.1), perform a detailed analysis of the behavior of our various models by comparing the types of errors made by learned versus hard-coded attention (section 6.2), and also examine different attention configurations that naturally follow from our experiments (section 6.3). ## related work attention mechanisms were first introduced to augment vanilla recurrent models (kalchbrenner and blunsom, 2013; sutskever et al., 2014; bahdanau et al., 2015; luong et al., 2015; chorowski et al., 2015; wu et al., 2016; miceli barone et al., 2017) but have become the featured component of the state-of-the-art transformer architecture (vaswani et al., 2017) for nmt. we review recent research that focuses on analysing and improving multiheaded attention, and draw connections to our work. the intuitive advantage of mha is that different heads can focus on different types of information, all of which will eventually be helpful for translation. voita et al. (2019) find that some heads focus on adjacent tokens to the query (mirroring our analysis in section 2), while others focus on specific dependency relations or rare tokens. correia et al. (2019) discover that some heads are sensitive to subword clusters or interrogative words. tang et al. (2018) shows that the number of mha heads affects the ability to model long-range dependencies. michel et al. (2019) show that pruning many heads from a pretrained model does not significantly impact bleu scores. similarly, voita et al. (2019) prune many encoder self-attention heads without degrading bleu, while tang et al. (2019) further simplify the transformer by removing the entire encoder for a drop of three bleu points. in contrast to existing literature on model pruning, we train our models without learned attention heads instead of removing them post-hoc. there have been many efforts to modify mha in transformers. one such direction is to inject linguistic knowledge through auxiliary supervised tasks (garg et al., 2019; pham et al., 2019). other work focuses on improving inference speed : yang et al. (2018) replace decoder self-attention with a simple average attention network, assigning equal weights to target-side previous tokens. @xcite wu et al. (2019) also speed up decoding by replacing selfattention with convolutions that have time-step dependent kernels; we further simplify this work with our fixed convolutional kernels in section 6. cui et al. (2019) also explore fixed attention while retaining some learned parameters, and vashishth et al. (2019) show that using uniform or random attention deteriorates performances on paired sentences tasks including machine translation. other work has also explored modeling locality (shaw et al., 2018; yang et al., 2018). ## conclusion in this paper, we present "hard-coded" gaussian attention, which while lacking any learned parameters can rival multi-headed attention for neural machine translation. our experiments suggest that encoder and decoder self-attention is not crucial for translation quality compared to cross attention. we further find that a model with hard-coded selfattention and just a single cross attention head performs slightly worse than a baseline transformer. our work provides a foundation for future work into simpler and more computationally efficient neural machine translation.