The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021)

Abstract: Mechanisms for encoding positional information are central for transformer-based language models. In this paper, we analyze the position embeddings of existing language models, finding strong evidence of translation invariance, both for the embeddings themselves and for their effect on self-attention. The degree of translation invariance increases during training and correlates positively with model performance. Our findings lead us to propose translation-invariant self-attention (TISA), which accounts for the relative position between tokens in an interpretable fashion without needing conventional position embeddings. Our proposal has several theoretical advantages over existing position-representation approaches. Proof-of-concept experiments show that it improves on regular ALBERT on GLUE tasks, while only adding orders of magnitude less positional parameters.

## introduction the recent introduction of transformer-based language models by @xcite has set new benchmarks in language processing tasks such as machine translation @xcite @xcite ), question answering @xcite , and information extraction @xcite . however, because of the non-sequential and positionindependent nature of the internal components of transformers, additional mechanisms are needed to enable models to take word order into account. @xcite identified three important criteria for ideal position encoding: approaches should be inductive, meaning that they can handle sequences and linguistic dependencies of arbitrary length, data-driven, meaning that positional dependencies are learned from data, and efficient in terms of the number of trainable parameters. separately, @xcite argued for translation-invariant positional dependencies that depend on the relative distances between words rather than their absolute positions in the current text fragment. it is also important that approaches be parallelizable, and ideally also interpretable. unfortunately, none of the existing approaches for modeling positional dependencies satisfy all these criteria, as shown in this paper makes two main contributions: first, in sec. 3, we analyze the learned position embeddings in major transformer-based language models. second, in sec. 4, we leverage our findings to propose a new positional-dependence mechanism that satisfies all desiderata enumerated above. experiments verify that this mechanism can be used alongside conventional position embeddings to improve downstream performance. our code is available. ## background transformer-based language models @xcite have significantly improved modeling accuracy over previous state-of-the-art models like elmo @xcite . however, the nonsequential nature of transformers created a need for other mechanisms to inject positional information into the architecture. this is now an area of active research, which the rest of this section will review. the original paper by @xcite proposed summing each token embedding with a position embedding, and then used the resulting embedding as the input into the first layer of the model. bert @xcite reached improved performance training data-driven d-dimensional em-induct-data-parameter translation parallel interpret-method ive? driven? efficient? invariant? -izable? able? sinusoidal position embedding @xcite absolute position embedding @xcite relative position embedding @xcite t5 @xcite flow-based @xcite synthesizer @xcite untied positional scoring @xcite rotary position embedding @xcite translation-invariant self-attention (proposed) the different position-representation approaches are summarized in ## analysis of existing language models in this section, we introspect selected high-profile language models to gain insight into how they have learned to account for the effect of position. ## proposed self-attention mechanism we now introduce our proposal for parameterizing the positional contribution to self-attention in an efficient and translation-invariant manner, optionally removing the position embeddings entirely. ## experiments the main goal of our experiments is to illustrate that tisa can be added to models to improve their performance ( results for tisa alone, in interestingly, the albert baseline on sts-b in ## conclusion we have analyzed state-of-the-art transformerbased language models, finding that translationinvariant behavior emerges during training. based on this we proposed tisa, the first positional information processing method to simultaneously satisfy the six key design criteria in