The Locality and Symmetry of Positional Encodings (2023)

Abstract: Positional Encodings (PEs) are used to inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models is not fully understood, especially given recent findings that various positional encodings are insensitive to word order. In this work, we conduct a systematic study of positional encodings in Bidirectional Masked Language Models (BERT-style) , which complements existing work in three aspects: (1) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry; (2) We show that the two properties are closely correlated with the performances of downstream tasks; (3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly. We believe that these results are the basis for developing better PEs for transformer-based language models.

## introduction transformer-based language models with positional encodings (pes) can improve performance considerably across a wide range of natural language understanding tasks. existing work resort to either fixed @xcite @xcite or learned @xcite @xcite ) pes to infuse order information into attention-based models. to understand how pes capture word order, prior studies apply visualized @xcite ) and quantitative analyses @xcite to various pes, and their findings conclude that all encodings, both human-designed and learned, exhibit a consistent behavior: first, the position-wise weight matrices show that non-zero values gather on local adjacent positions. second, the matrices are highly symmetrical, as shown in figure 1 . these @xcite . all matrices highly attend to local positions (locality) and are nearly symmetrical (symmetry). in this work, we focus on uncovering the core properties of pes in bidirectional masked language models (bert-style @xcite ). we do not include analyses of @xcite and llama @xcite since there is an obvious distinction between them and bert-style models. decoder-only models predict the next word based on previous words in the sentence (left-to-right mode), and therefore implicitly introduce directional information, which is different from bidirectional models. furthermore, recent studies have shown that decoderonly transformers without positional encodings are able to achieve competitive or even better performance than other explicit positional encoding methods @xcite . hence, we focus on the bert-style models in this work leaving the decode-only models as a future study. we study various positional encodings by introducing two quantitative metrics, locality and symmetry. our empirical studies demonstrate that the two properties are correlated with sentence representation capability. this explains why fixed encodings are designed to satisfy them and learned encodings are favorable to be local and symmetrical. moreover, we show that if attention-based models are initialized with pes that already share good locality and symmetry, they can obtain better inductive bias and significant improvements across 10 downstream tasks. our findings of the local preference may explain why sliding window attentions @xcite and attention sinks @xcite can work effectively. although pes with locality and symmetry can achieve promising results on natural language understanding tasks (such as glue @xcite ), the symmetry property itself has an obvious weakness, which is not revealed by previous work. existing studies use shuffled text to probe the sensitivity of pes to word orders @xcite @xcite @xcite , and they all assume that the meaning of sentences with random swaps remains unchanged. however, the random shuffling of words may change the semantics of the original sentence and thus cause the change of labels. for example, the sentence pair below from snli @xcite satisfies the entailment relation: a. a man playing an electric guitar on stage b. a man playing guitar on stage if we change the word order of the premise sentence so that it becomes "an electric guitar playing a man on stage", a fine-tuned bert still finds (incorrectly!) that the premise entails the hypothesis. starting from this point, we design two new probing tasks of word swap: constituency shuffling and semantic role shuffling. the former preserves the original semantics of the sentence by swapping words inside constituents (local structure) while the latter intentionally changes the semantics by swapping the semantic roles in a sentence (global structure), i.e., the agent and patient. our results show that existing language models with various pes are robust against local swaps, but extremely fragile against global swaps. the key contributions of our work are: ## preliminaries the central building block of transformer architectures is the self-attention mechanism @xcite . given an input sentence: @xmath0, x 2 , ..., x n } ∈ r n×d , where n is the number of words and d is the dimension of word embeddings, the attention computes the output of the i-th token as: self-attention heads do not intrinsically capture the word order in a sequence because there is no positional constraint in equation 1. therefore, specific methods are used to infuse positional information into self-attention @xcite . ## conclusion we have proposed a series of probing analyses for understanding the role of positional encodings in sentence representations. we find two main properties of existing encodings, locality and symmetry, which are correlated with the performance of downstream tasks. we first investigate the linguistic role of positional encodings in sentence representation. meanwhile, we point out an obvious flaw of the symmetry property. we hope that these findings will inspire future work to better design positional encodings.