Sparse Attention with Linear Units (2021)

Abstract: Recently, it has been argued that encoder-decoder models can be made more interpretable by replacing the softmax function in the attention with its sparse variants. In this work, we introduce a novel, simple method for achieving sparsity in attention: we replace the softmax activation with a ReLU, and show that sparsity naturally emerges from such a formulation. Training stability is achieved with layer normalization with either a specialized initialization or an additional gating function. Our model, which we call Rectified Linear Attention (ReLA), is easy to implement and more efficient than previously proposed sparse attention mechanisms. We apply ReLA to the Transformer and conduct experiments on five machine translation tasks. ReLA achieves translation performance comparable to several strong baselines, with training and decoding speed similar to that of the vanilla attention. Our analysis shows that ReLA delivers high sparsity rate and head diversity, and the induced cross attention achieves better accuracy with respect to source-target word alignment than recent sparsified softmax-based models. Intriguingly, ReLA heads also learn to attend to nothing (i.e. ‘switch off’) for some queries, which is not possible with sparsified softmax alternatives.

## introduction attention models @xcite have been hugely successful recently, with transformer @xcite in particular, advancing state of the art on various tasks, such as machine translation @xcite , document summarization @xcite and speech processing @xcite , and delivering a large impact on a broad range of nlp tasks via large-scale self-supervised pretraining @xcite . 1 source code is available at https://github.com/ bzhanggo/zero . at the core of attention is a mechanism that dynamically highlights relevant context features for a given query input. in the vanilla softmax-based attention model @xcite , this is achieved by imposing a categorical distribution constraint on the query-context relevance (i.e. attention) scores, implemented with the softmax activation (see figure 1(a) ). smatt produces dense distributions, assigning some small amounts of attention even to irrelevant features. this complicates the analysis of the information flow in the model, and has led researchers to study sparse alternatives, which often lead to improved model performance and/or interpretability @xcite . efforts in this category include designing fixed sparsity patterns @xcite and creating sparsified softmax variants @xcite . however, these methods also have drawbacks. fixed sparsity patterns lack flexibility and generalize poorly across tasks. sparsified softmax variants often depend on complex inference algorithms (e.g., requiring the sorting operation), which reduces their efficiency. in this paper, we propose rectified linear attention (rela) to alleviate the above problems. rela uses relu rather than softmax as an activation function for attention scores, abandoning the probabilistic constraint. 2 relu is inherently sparse since negative activations are dropped, and we will show that such sparse behaviour indeed emerges during training. in contrast to softmax activations, the output of relu can be any non-negative value, providing extra flexibility. to stabilize gradients and ease model convergence, we apply layer normalization together with a specialized initialization or a gating mechanism. figure 1(b) shows rela, and also contrasts it with smatt. rela is an easy-to-implement drop-in replacement for smatt that requires no specialized operations or inference processes. note that the behaviour of rela is data-driven, and it does not enforce a constant attention mass or sparsity level across queries, even allowing for null attention (all attention scores are zero) for some queries. we provide experimental results for rela with transformer on five machine translation tasks, along with an in-depth analysis on wmt14 english-german task. our contributions are summarized below: ## related work rela ensures sparsity in attention. an alternative solution in this direction is to develop sparsified softmax alternatives, such as sparsemax @xcite , entmax @xcite , fusedmax @xcite , and hashing/clustering-based variants @xcite . these models often require dedicated algorithms for forward and backward propagation, at the cost of a significant computational overhead. another strategy is to manually define sparse patterns inspired by task-specific attention analysis. raganato et al. (2020) corroborated the feasibility of fixed patterns for transformer encoder in translation. child et al. (2019) introduced local and strided patterns to scale smatt up to very long inputs. unlike data-driven approaches, whether these patterns could generalize to different tasks and settings is still an open question. in contrast, rela is both data-driven and efficient. in this respect, our work shares similarity with the explicit sparse transformer @xcite which also delivers faster speed but still depends on top-k sorting as in sparsemax and entmax with k, a tunable hyperparameter. note that all the above mentioned methods follow the categorical distribution constraint on attentions, while rela goes beyond. thus, unlike rela, none of them enables null attentions. a different type of linear attention model is proposed by @xcite and @xcite , who aim at reducing the o(n 2 ) complexity in smatt. these models behave fundamentally differently from rela, because they eliminate the token-wise modeling rather than introducing sparsity. the explanatory power of standard attention weights is hotly debated @xcite . much of the criticism stems from the observation that low attention scores do not always imply irrelevance of the corresponding feature, as the information can still flow and its influence can be large (e.g., due to the large magnitude of the corresponding features). in contrast, sparse variants, including rela, assign exact zeroes, ensuring that the information flow from the corresponding features within the attention component is cut completely. even with standard attention, prior studies show some evidence that attention partially reflects linguistic properties. in machine translation, the encoder-decoder attention captures the source-target word alignment to a certain degree @xcite , with recent work further strengthening this via specific induction methods @xcite @xcite . we apply analysis techniques from previous work to analyze our models. ## background: attention in transformer many variants of attention mechanism have been developed since its first proposal @xcite . in this paper, we focus on the one used by transformer, namely multihead scaled dot-product attention (mhatt), in an encoder-decoder setup. given query inputs @xmath0, each head in mhatt summarizes query-relevant context information as follows: with @xmath1; k, @xmath2, yw v , where n and m are the query and context length, respectively; d and d h are the model and head dimension, respectively; w * ∈ r d×d h denotes trainable model parameters. α ∈ r n×m is the attention weight, which estimates the degree of relevance between one query input and each context. the softmax normalizes the scores and ensures that the attention weights α define a categorical distribution. f (•) is a scoring function. different attention mechanisms make different choices for f (•), but the use of softmax, or its sparsified variants, is universal. smatt in transformer adopts the scaled dot product for f (•), which is further extended by mhatt to allow for parallel attentions in different sub-spaces over the same inputs: in the encoder-decoder framework, mhatt is used in three different ways: encoder attention, decoder attention and cross attention, modeling intra-source, intra-target, and source-target dependencies, respectively. transformer performs layered mhatt with residual connection and layer normalization @xcite to handle variations of token-wise dependencies. the learning of mhatt is guided by the training objective, often without direct supervision. ## rectified linear attention we argue that the use of the softmax function in smatt (eq. 1) has two undesirable consequences: both potentially hamper interpretability and even performance. 3 as an alternative to sparsified softmax variants @xcite , we go one step further and consider whether the softmax, or broadly the categorical distribution, could be avoided completely. ## conclusion and future work in this paper, we have presented rectified linear attention (rela), a novel softmax-free sparse attention model. rela avoids the categorical distribution assumption for attention, and, due to using relu as activation function, prunes out all negative attention scores and produces sparse attention. to stabilize model training, we add a normalization layer to attention outputs with a specialized initialization or gating structure. rela is datadriven, computationally efficient and is a drop-in replacement of smatt. experiments on five machine translation tasks with transformer demonstrate rela's effectiveness in delivering comparable translation quality to softmax-based baselines. results also show that rela is substantially faster than sparsemax and 1.5-entmax at training and decoding. the attentions learned by rela correspond better to word alignment, with high head diversity and sparsity rate. also, in contrast to alternative sparse attention approaches, rela pro-duces null attentions, i.e. a head can assign a total attention of zero for some queries, leading to highly specialized attention heads and showing potential to indicate translation quality. in the future, we will apply rela to other neural models and tasks. we are interested in scaling rela to very long inputs @xcite , or multi-source architectures where the relevance of each source may vary. in its current formulation, the sparsity level of rela emerges from the threshold in relu which prunes negative scores. we are interested in ways to manipulate the level of sparsity, or make the threshold differentiable. head 1 h o w e v e r , in a n in te rv ie w , b lo @ @ o m h a s sa id th a t h e a n d k e r@ @ r st il l lo v e e a ch o th e r . e o s head 2 h o w e v e r , in a n in te rv ie w , b lo @ @ o m h a s sa id th a t h e a n d k e r@ @ r st il l lo v e e a ch o th e r . e o s head 3 h o w e v e r , in a n in te rv ie w , b lo @ @ o m h a s sa id th a t h e a n d k e r@ @ r st il l lo v e e a ch o th e r . e o s in e in e m in te rv ie w sa g te b lo @ @ o m je d o ch , d a ss e r u n d k e r@ @ r si ch n o ch im m e r li e b e n . e o s head 4 h o w e v e r , in a n in te rv ie w , b lo @ @ o m h a s sa id th a t h e a n d k e r@ @ r st il l lo v e e a ch o th e r . e o s head 5 h o w e v e r , in a n in te rv ie w , b lo @ @ o m h a s sa id th a t h e a n d k e r@ @ r st il l lo v e e a ch o th e r . e o s head 6 h o w e v e r , in a n in te rv ie w , b lo @ @ o m h a s sa id th a t h e a n d k e r@ @ r st il l lo v e e a ch o th e r . e o s head 7 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s s ch a u sp ie le r o r@ @ la n d @ @ o b lo @ @ o m u n d m o d e l m ir @ @ a n d a k e r@ @ r w o ll e n k ün ft ig g e tr e n n te w e g e g e h e n . e o s head 0 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s head 1 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s head 2 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s head 3 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s s ch a u sp ie le r o r@ @ la n d @ @ o b lo @ @ o m u n d m o d e l m ir @ @ a n d a k e r@ @ r w o ll e n k ün ft ig g e tr e n n te w e g e g e h e n . e o s head 4 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s head 5 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s head 6 a c @ @ to rs o r@ @ la n d @ @ o b lo @ @ o m a n d m o d e l m ir @ @ a n d a k e r@ @ r w a n t to g o th e ir se p a ra te w a y s . e o s head 7 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s m ir @ @ a n d a k e r@ @ r u n d o r@ @ la n d @ @ o b lo @ @ o m si n d e lt e rn d e s z w e i@ @ jä h ri g e n f l@ @ y n n . e o s head 0 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s head 1 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s head 2 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s head 3 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s m ir @ @ a n d a k e r@ @ r u n d o r@ @ la n d @ @ o b lo @ @ o m si n d e lt e rn d e s z w e i@ @ jä h ri g e n f l@ @ y n n . e o s head 4 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s head 5 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s head 6 m ir @ @ a n d a k e r@ @ r a n d o r@ @ la n d @ @ o b lo @ @ o m a re p a re n ts to tw o @ -@ y e a r @ -@ o ld f l@ @ y n n . e o s head 7 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s in e in e m in te rv ie w m it u s @ -@ j o u rn @ @ a li @ @ st i@ @ n k @ @ a ti e c o u @ @ ri c , d a s a m f re it a g ( o rt s@ @ z e it ) a u sg e @ @ st ra h lt w e rd e n so ll te , sa g te b lo @ @ o m , " d a s l e b e n v e rl äu ft m a n ch m a l n ic h t g e n a u so , w ie w ir e s p la n e n o d e r e r@ @ h o ff e n " . e o s head 0 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s head 1 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s head 2 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s head 3 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s in e in e m in te rv ie w m it u s @ -@ j o u rn @ @ a li @ @ st i@ @ n k @ @ a ti e c o u @ @ ri c , d a s a m f re it a g ( o rt s@ @ z e it ) a u sg e @ @ st ra h lt w e rd e n so ll te , sa g te b lo @ @ o m , " d a s l e b e n v e rl äu ft m a n ch m a l n ic h t g e n a u so , w ie w ir e s p la n e n o d e r e r@ @ h o ff e n " . e o s head 4 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s head 5 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s head 6 in a n in te rv ie w w it h u s jo u rn a li st k @ @ a ti e c o u @ @ ri c , w h ic h is to b e b ro a d c a st o n f ri d a y ( lo c a l ti m e ) , b lo @ @ o m sa id , " so m e ti m e s li fe d o e sn 't g o e x a c tl y a s w e p la n o r h o p e fo r " . e o s head 7