The Curious Case of Absolute Position Embeddings (2022)

Abstract: Transformer language models encode the notion of word order using positional information. Most commonly, this positional information is represented by absolute position embeddings (APEs), that are learned from the pretraining data. However, in natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been studied. In this work, we observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information. Specifically, when models are subjected to sentences starting from a non-zero position (excluding the effect of priming), they exhibit noticeably degraded performance on zero- to full-shot tasks, across a range of model families and model sizes. Our findings raise questions about the efficacy of APEs to model the relativity of position information, and invite further introspection on the sentence and word order processing strategies employed by these models.

## introduction recently, transformer @xcite language models (tlms) have been widely used for natural language applications. such models incorporate positional encodings: vectors encoding information about the order of words in context. many models, such as roberta @xcite , @xcite and opt @xcite , utilize absolute position embeddings (apes) that directly encode absolute (linear) word order. apes appear to contribute to the performance of such models; although when they are removed, some models become sensitive to ablative word scrambles @xcite , while others work optimally @xcite . thus, what precisely apes contribute remains unclear. ## approach position encodings used by tlms come in three broad categories: fixed sinusoidal embeddings as proposed by @xcite , absolute or learned popularized by bert @xcite family of masked language models, and relative positions @xcite used by t5 @xcite . @xcite presents a comprehensive overview of current encoding strategies. despite being an older method, absolute positional embeddings (apes) are reportedly better than its relative counterparts on several tasks @xcite , and are still used by majority of the large pre-trained tlms, including the recently released opt @xcite . apes compute token representation after adding the input token to the position embedding for the corresponding position: tlms offer various sizes of context window, which is the maximum sequence length in tokens it can train and infer on. since this context window is usually larger than the average sentence length, multiple sentences can be packed together to "fill" the context window during pre-training. this allows tlms to learn that sentences can start from various positions in their context window. if models trained with apes do encode relativity of position, then the sentence representations should be roughly equal throughout the context window, regardless of their starting position. ## impact of phase shifts on in-context learning more recently, zero-shot and few-shot inference, commonly referred to as in-context learning, have become a de facto standard in evaluating pretrained language models @xcite . in this approach, the model's predictions are produced by conditioning it on certain prompts, such as instructions (zero-shot setting) or a few examples of inputoutput pairs (few-shot setup). in both cases, the model faces an extended input text, and we suspect it will be affected by deficiencies of ape. to evaluate this hypothesis, we employ an experimental setup similar to §3. under zero-shot and five-shot inference regimes, we assess the model performance on standard nlp tasks when it is fed with inputs in increasing values of phase shifts. we choose opt model family, because it is available in a wide range of sizes (125m to 30b parameters), allowing allows us to examine the behavior of ape at different scales. moreover, our evaluations take into account four tasks reported in the original pa- in most tasks, the performance deteriorates when the model process inputs in any other phase shift than zero, especially in zero-shot inference. more importantly, the model's performance is not always adversely affected by phase shifts. in fact, figure 5 shows that non-zero starting positions result in the best accuracy for many prompts. this erratic performance is present in all model sizes, and scaling the number of parameters does not help. furthermore, one can see larger models are more affected by shifted starting position, which suggests that absolute positional embedding might need more data or training as the number of parameters increases. ## impact of phase-shifts on fine-tuning finally, we investigate the effect of phase shift in fine-tuning. we ask whether the models can generalize to out-of-phase sentences for a given task. we train roberta, bart, gpt2 and opt models on cola, rte and mrpc tasks from the glue benchmark @xcite and evaluate them on phase-shifts. we choose these three relatively small tasks in order to decrease the number of gradient updates to position embeddings during fine-tuning. we perform a cross-phase analysis by training and evaluating across different phase shifts (@xmath0, 100, 200, 300) for all models on the same set of datasets, and show the averaged performance. we observe for all models, the task performance drops during out-of-phase evaluation (non-diagonals in figure 6 ). the drop in performance of evaluating out-ofphase sentences might just be simply attributed to overfitting on position information during finetuning. however, we observe that for all tasks, training and evaluating on the same phase-shift is worse when k ̸ = 0 (diagonals in figure 6 ). out-ofphase training appears to be worst for cola, which suffers drastically when fine-tuning on different phase shifts. these results highlight a potential task data bias with respect to different positions. ## conclusion in this work, we investigate the abilities of apes in encoding the relative positions of the tokens in an input. we observe that tlms using apes encode sentences differently based on the starting position of the sentence in the context window. this result has major implications in the way we perceive the sentence processing capabilities of tlms. specifically, we observe that the representation of the same sentence varies depending on where it is in the context window, such that it impacts zero shot, few shot and full shot task performance of sub-window sentences. future work could leverage the start position in building robust and positiongeneralizable models. we hope our work can inform the community on the pitfalls of using apes, and inspire development and adoption of alternative relative position embedding based approaches.