Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding (2024)

Abstract: Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they can not perform **length extrapolation** to handle long sequences. Thus, numerous methods have emerged to enhance the length extrapolation of Transformers. Despite the great research efforts, a systematic survey is still lacking. To fill this gap, we delve into these advances in a unified notation from the perspective of positional encoding (PE), as it has been considered the primary factor on length extrapolation. Specifically, we begin with extrapolatable PEs that have dominated this research field. Then, we dive into extrapolation methods based on them, covering position interpolation and randomized position methods. Finally, several challenges and future directions in this area are highlighted. Through this survey, We aim to enable the reader to gain a deep understanding of existing methods and provide stimuli for future research.

## introduction it has been suggested that with limited learning resources, humans can potentially comprehend utterances of infinite length by understanding their components and structures @xcite . in natural language processing (nlp), given the limited training data @xcite and compute, models cannot learn from large-scale long sequences and thus are also expected to possess such generalization ability to process long sequences @xcite . however, it is a challenging task for the de facto * corresponding author transformer architecture @xcite , though transformer-based large language models (llms) @xcite have drastically advanced the nlp field. transformer-based models are trained on sequences with a maximum length @xcite @xcite , as a result of the quadratic memory and computational complexity with regard to input length. to make matters worse, some research reveals that transformers might have gained their performance from surface-level memorization instead of abstract, generalizable skills @xcite , which means they can hardly break through the maximum training length and perform poorly on sequences with length beyond it @xcite , i.e., they cannot perform length extrapolation @xcite . to offer a more comprehensive understanding of the challenges in length extrapolation, we present comparison results of three state-of-the-art models with different context sizes on several generation tasks in appendix a.1. the length limit together with poor length extrapolation prevents llms from handling long sequences, such as dna and protein sequences @xcite , high-resolution images @xcite , and even videos @xcite . moreover, existing approaches for harnessing the full potential of llms also demand a larger context window, to incorporate elaborate prompts @xcite , sufficient in-context demonstrations @xcite and long-term memory of agents @xcite . hence, there is a growing body of research trying to strengthen length extrapolation of llms @xcite @xcite @xcite , mostly from the perspective of positional encoding @xcite . despite the prosperity in this area, a systematic survey is still lacking. we aim to fill this blank by investigating existing approaches that enable and enhance length extrapolation of transformers. specifically, a brief formal introduction to transformer is given in §2 as a solid foundation for further discussion. then, we comprehensively summarize extrapolatable pes proposed from the birth of transformer to the prevalence of llms in §3. note that we focus exclusively on pes proposed for better extrapolation and omit others, since there is already an insightful survey on pes of transformer @xcite . based on these pes, many novel methods emerge in the era of llms to further enhance extrapolation, which we intentionally centralize in §4, covering popular position interpolation methods and randomized methods. these advancements demonstrate the vibrancy and vastness of this area, from which we distill future directions and insights, represented in §5 and §6. ## preliminary in this section, we follow @xcite to present a formal description of the encoder layer of the transformer, as the decoder layer is almost the same except for the cross-attention mechanism. given an input matrix @xmath0, an encoder layer f : r n×d -→ r n×d with f (x) = z is defined by: note that in the above descriptions, we have not imposed any limit on input length n, which means the transformer is naturally equipped with a notion of length extrapolation. theoretically, a fixed setting of transformer weights defines a sequence-tosequence function on sequences of arbitrary length @xcite . if the function applies the correct transformation for inputs of any length, it is expected to length extrapolate @xcite . however, we have to break this nature by integrating pe with transformers to inject position information into them. otherwise, they are permutation equivalent or order invariant foot_1 . thus, pes are central to length extrapolation and form the core focus of this survey. ## extrapolatable positional encodings sinusoidal position embeddings are proposed with transformer as it may help extrapolate to longer sequences beyond training @xcite . the idea behind this claim, that length extrapolation can be enabled by simply changing pe, has been widely supported and demonstrated @xcite @xcite . hence, developing better pes has pe manifestation learnable integration injection layer ape sinusoidal @xcite embedding ✕ add initial with shift invariance shape @xcite embedding ✕ add initial cape @xcite embedding ## extrapolation methods in llms era based on pes in §3, various methods have been developed to further enhance length extrapolation of llms. this section is separated in response to this wave, focusing on interpolation methods and randomized pes, as illustrated in figure 3 . ## discussions 6.1 length-extrapolated and long-context transformers throughout this survey, we position length extrapolation as a promising avenue towards long-context transformers. however, as stated in §1, it's the length limit and poor length extrapolation together that prevents transformers from processing long sequences, thus the more direct way to extend the context window is to simply relax the length limit. the most intuitive way to achieve large context window is directly pre-training the model or fine-tuning (continual pre-training) a pre-trained model on long sequences. @xcite empirically demonstrated that long context continual pre-training is more efficient and similarly effective compared to pre-training from scratch with long sequences. however, both pre-training and fine-tuning (continual pre-training) are costly and demand large-scale high-quality long data, which is scarce @xcite . to reduce memory and computational overhead during training, recurrent transformer variances integrate recurrence with attention @xcite while efficient transformer variants @xcite ) mainly aim at improving the quadratic complexity of attention mechanism, but both usually compromise some of the modeling capability and still need large-scale long sequence data. flash attention @xcite greatly improves both training and inference efficiency of transformers with little to no overhead, leading to models with much larger context window @xcite @xcite . on the other side, there are more radical research efforts that attempt to abandon attention and its quadratic complexity with regard to sequence length completely, such as s4 @xcite , rwkv @xcite , and hyena @xcite . further, some recent studies have attempted to scale these novel architectures to billions of parameters, leading to the emergence of mamba @xcite and rwkv-5/6 @xcite . however, it has been demonstrated that transformer models perform dramatically better than state space models like s4 at copying and retrieving information from context @xcite . thus, whether these novel architectures are better than transformer and how they perform on real-world scenarios remains to be evaluated. ## conclusion through this survey, we systematically summarized existing methods and recent advances in length extrapolation from the perspective of pe. specifically, we meticulously categorize extrapolatable pes and further dive into methods based on these pes in llms era. in addition, we highlight existing challenges and identify new trends in this research field, hoping to facilitate researchers and provide stimuli for future research.