Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention (2020)

Abstract: Self-Attention Networks (SANs) are an integral part of successful neural architectures such as Transformer (Vaswani et al., 2017), and thus of pretrained language models such as BERT (Devlin et al., 2019) or GPT-3 (Brown et al., 2020). Training SANs on a task or pretraining them on language modeling requires large amounts of data and compute resources. We are searching for modifications to SANs that enable faster learning, i.e., higher accuracies after fewer update steps. We investigate three modifications to SANs: direct position interactions, learnable temperature, and convoluted attention. When evaluating them on part-of-speech tagging, we find that direct position interactions are an alternative to position embeddings, and convoluted attention has the potential to speed up the learning process.

## introduction self-attention mechanisms are at the core of successful neural network architectures in natural language processing (e.g., transformer by @xcite ). compared to recurrent neural networks they do not have an inherent sequential bias, yet they allow the network to transfer knowledge across a sequence of length t in a constant number of steps. typically, a self-attention layer consists of multiple attention heads and is itself part of a more sophisticated layer such as a transformer encoder block. we propose three minor modifications to the self-attention mechanism: (1) position embeddings @xcite are used to inject positional information into sans. we argue that learning position interactions can be modeled more directly than learning separate position embeddings and propose to replace embeddings with a direct position interaction matrix. (2) we hypothesize that spiky distributions generated by a softmax function within the attention head hinders the network from considering the broader sentence context effectively. thus we introduce additional scalar parameters, a learnable temperature, that can support the network in using the context more effectively. we perform experiments on part-of-speech (pos) tagging. we argue that a pos model can only be successful for ambiguous and out-of-vocabulary tokens if it carefully considers and processes the context. thus we consider pos a suitable task to probe whether our modifications on the attention matrix enable more efficient learning. we perform experiments on the penn treebank (ptb) and on 47 languages of universal dependencies (ud). in short, our findings are: (i) modeling absolute and relative position information through direct interaction matrices is a feasible alternative to position embeddings. (ii) learnable temperature has almost no effect besides a small increase for out-of-vocabulary tokens. (iii) convoluted attention achieves a higher accuracy after fewer epochs (more efficient learning) on ptb and has higher performance on ud. while results for convoluted attention are promising we are aware that only evaluating on pos is a very restricted setting. thus we plan to extend this study in future work. this work is licensed under a creative commons attribution 4.0 international licence. licence details: http:// creativecommons.org/licenses/by/4.0/ . word embeddings position embeddings character embeddings cnn (3 layers) multi-head self-attention softmax pos predictions multi-head self-attention + + dropout dropout concatenate add / concatenate concatenate max-pooling (a) model architecture. for simplicity only 2 self-attention layers are shown. hyperparameter ptb ud # layers 4 char cnn filter width 3 # char cnn filters 64 # attention heads 4 embedding dimension d 300 128 position emb. dim. 300 128 character emb. dim. 64 finetune emb. no yes use pretrained emb. yes no finetune position emb. yes finetune character emb. yes vocabulary size 20000 50% of unique words max sequence length t 60 max char seq. length 20 early stopping yes, 3 epochs patience activation function relu dropout rate 0.1 optimizer rmsprop with default parameters as in keras.io/api/optimizers (b) model hyperparameters. san +pe[add] +p +r +temp +conv +conv2d 7,691,823 +18,000 +14,400 +480 +48 +173,760 +160 (c) number of parameters for ptb when adding each extension to san. +r, +temp and +conv2d only add very few parameters to the model. san+p has less parameters than san+pe[add] when t 2 × @xmath0. ## related work many variants of positional embeddings have been explored: @xcite reported on sinusoidal and learned position embeddings, @xcite explored relative position embeddings and @xcite introduced directional self-attention. we propose to replace traditional position embeddings by a direct position interaction matrix. recently @xcite proposed to model relative positions with scalar values, an idea also investigated by @xcite . this approach is similar to our san+r. contemporary to this submission, @xcite proposed tupe, which is similar to san+p. they also find it to be a feasible alternative to position embeddings and report slight performance increases. in contrast to weight normalization @xcite , a related method to learnable temperature, we do not normalize the weight matrices. instead we only add a learnable scalar parameter and observed that normalizing the weights actually harms performance. @xcite introduced a self-adaptive temperature. however, they focused on parametrizing the temperature of timestep t using the activations from timestep t-1. contemporary to this work, @xcite proposed query-key normalization in transformers. there is range of work trying to combine attention with convolution @xcite . we are not aware of any work that applies convolution directly to attention weights. ## conclusion we conclude that position embeddings can be replaced with direct position interactions. 2 learnable temperature has almost no effect. convoluted attention speeds up learning on ptb and yields better results on ud. we are aware that this paper is a small study with limited validity as it considers only one task. given that convoluted attention yielded promising results, we plan to extend this line of experiments to additional tasks and architectures in future work. our code is available. 3