Recurrent Attention for Neural Machine Translation (2021)

Abstract: Recent research questions the importance of the dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. In this paper, we push further in this research line and propose a novel substitute mechanism for self-attention: Recurrent AtteNtion (RAN) . RAN directly learns attention weights without any token-to-token interaction and further improves their capacity by layer-to-layer interaction. Across an extensive set of experiments on 10 machine translation tasks, we find that RAN models are competitive and outperform their Transformer counterpart in certain scenarios, with fewer parameters and inference time. Particularly, when apply RAN to the decoder of Transformer, there brings consistent improvements by about +0.5 BLEU on 6 translation tasks and +1.0 BLEU on Turkish-English translation task. In addition, we conduct extensive analysis on the attention weights of RAN to confirm their reasonableness. Our RAN is a promising alternative to build more effective and efficient NMT models.

## introduction transformer models have achieved remarkable success in neural machine translation (nmt) @xcite @xcite . one of the most crucial component of transformer is the dot-product multi-head selfattention, which is essential to learn relationships between words as well as complex structural representations. however, many studies have shown that the pairwise self-attention is over-parameterized and leads to a costly inference @xcite @xcite . based on these observations, various improved networks are proposed by either pruning negligible heads @xcite or replacing selfattention with more efficient one @xcite @xcite . more recently, several researches take this direction to an extreme by replacing dot-product selfattention with fixed or trainable global positionbased attentions @xcite @xcite . for example, @xcite roughly modeled attention weights as hard-coded gaussian distributions, based on the observation that most heads only focus their attention on a local neighborhood. another more representative method is random synthesizer proposed by @xcite . different from @xcite , they simply treat attention weights of all heads in each layer as trainable parameters. at inference time, the attention weights are directly retrieved based on the index of the query without dot-product operation. however, these variants are not the ideal alternatives of selfattention due to the unsatisfactory performance. in this paper, we go further along this research line and show that self-attention is empirically replaceable. we propose a novel attention mechanism: recurrent attention (ran). specifically, ran starts with an unnormalized initial attention matrix for each head, which is randomly initialized and trained together with other model parameters. then we introduce a recurrent transition module, which takes the initial attention matrices as the input and refines them by layer-wise interaction between adjacent layers. the motivation of recurrent transition module is based on the observation that attention weights show a regular pattern and have certain correlation across layers @xcite . our ran not only discards the expensive pairwise dot-product of self-attention but also exploit correlation between attention weights of different layers, achieving a more efficient and compact nmt model. 1 to verify the effectiveness of ran, we conduct experiments on a wide range of translation tasks involving 10 language pairs. compared with a vanilla transformer, our ran shows competitive or better performance with lower latency and fewer parameters. we conduct extensive analysis on the learned ran weights showing that the learned attention pattern are reasonable and explainable, which gives credit for the improvement. ## our method in this section, we first give a brief introduction of self-attention and we refer readers to the original paper @xcite for details. then, we introduce the proposed recurrent attention mechanism in detail. ## experiment in this section, we evaluate ran on wmt and nist translation tasks including 10 different language pairs altogether. we apply ran to the encoder (ran-e), the decoder (ran-d), or both of them (ran-all), respectively. for baselines, we compare against the standard transformer (transf for short) @xcite , and two most related work that are hard-coded transformer (hc-sa) @xcite and random synthesizer (syn-r) @xcite . ## analysis in order to better understand ran, we conduct comprehensive empirical studies on its behavior on the wmt14 en⇒de test set. ## application to other generation tasks to investigate the generalization of ran, we conduct additional experiments on abstractive summarization using the cnn/dailymail dataset and dialogue generation using the personachat dataset. 8 on summarization task, all of the models are trained for 300k steps on 2 gpus with a batch size of 128 sentences. for dialog generation, we segment all dialog with bert tokenizer 9 and train a small transformer for 20k steps. we use nlg_eval 10 for evaluation and report the results in ## related work the introduction of attention mechanisms into nmt can be traced back to @xcite and @xcite , which are used to learn soft word alignments between language pairs. due to the significant improvements in translation quality, the attention models have become an critical component of nmt models. more recently, @xcite proposed transformer that achieved the state-of-the-art and soon becomes the most popular nmt architecture. the self-attention network (san), playing an important role in the transformer, has been investigated and analyzed by a number of recent studies @xcite ; correia 8 we directly use the dataset from https://github.com/paddlepaddle/research/tree/master/nlp /dialogue-plato 9 https://github.com/google-research/bert 10 https://github.com/maluuba/nlg-eval et al. @xcite @xcite . these studies have shown that transformer models are over-parametrized and the self-attention models learn redundant information that can be pruned in various ways. the observations motivate lots of attempts in improvement of san, including 1) improving its computation efficiency and 2) completely replacing it with fixed or learnable global attention patterns. for the former thread, several studies bias attention distributions towards more local areas @xcite @xcite or replace san with convolutional modules @xcite , which are more in line with the linguistic expectation. @xcite share attention weights in adjacent layers and enable efficient re-use of hidden states in a vertical manner. on the other hand, given that most attention heads learn simple, and often positional patterns, many researchers turn to substitute instance-wise self-attention with global position-based attention patterns. concretely, @xcite use average attention models in the decoder of transformer. you et al. (2020) model the attention distribution as hard-coded gaussian ones, and raganato et al. 2020 also replace all but one attention head of each encoder layer with totally position based attentive patterns. more recently, @xcite propose random synthesizer in which the attention matrices as trainable parameters that are random initialized and trained with other model parameters. overall, our work is related to the second type of approaches and most related to @xcite and @xcite . unlike @xcite applying hard-coded gaussian attention focusing on local windows, the ran can learn more flexible attention distribution. @xcite allocate different learnable attention matrix for every head in each layer. in addition, so many individual matrices are hard to train and do not reduce the overall parameters at all. in contrast, our ran uses the recurrent mechanism to refine the learnable attention matrices layer by layer to improve the model capacity, and has the advantages of saving parameters and modeling relationships of attention between adjacent layers. ## conclusion in this paper, we considered a simpler transformer architecture for nmt without costly dot-product self-attention. for this goal, a novel recurrent atten-tion mechanism (ran) is proposed, which takes the initial attention matrices as a whole and update it by a recurrent transition module recurrently. experiments on 10 representative translation tasks show effectiveness of ran. in the future, we will explore the application of ran on cross-attention.