PEPE : Long-context Extension for Large Language Models via Periodic Extrapolation Positional Encodings (2025)

Abstract: Long-context extension seeks to expand the contextual window in pre-trained large language models (LLMs), allowing them to handle several multiples of their original training context lengths. The primary method for extending the window length involves expanding the initial positional encodings, such as interpolating and extrapolation new positions based on Rotary Position Embedding (RoPE). This expansion inevitably disrupts the positional encodings learned during pre-training, thereby affecting the attention allotment and introducing unseen positional encoding distributions. To address this issue, we propose a new extension strategy based on RoPE, namely Periodic Extrapolation Positional Encodings (PEPE). This strategy expands pre-trained high dimensional components of positional encodings by replicating them in a periodic manner, thereby neither altering the learned positional encoding spaces nor introducing new positional encoding distributions. Experiments demonstrate that PEPE-based approaches can significantly improve long-context extension capabilities using just one-fourth the fine-tuning steps required by state-of-the-art methods. In addition, we analyze the characteristics of PEPE based methods and the key parameters that contribute to their effectiveness. The code is publicly available.

## introduction nowadays, transformer-based @xcite large language models (llms) have experienced rapid advancement @xcite , showcasing impressive reasoning abilities that have significantly propelled progress in the field of natural language processing. long-context extension @xcite is a fundamental research area in the field of llm, referring to the ability to process texts with substantially extended context lengths. extending the context window of llms basically requires augmenting the positional encodings, a critical step in improving the model's capability to handle longer sequences effectively. rotary position embeddings (rope) @xcite have emerged as the leading positional encoding method for llms, as seen in widely used models such as llama @xcite , palm @xcite and qwen @xcite . nevertheless, when the context length surpasses the length of the training data, a straightforward extrapolation based on rope fails to achieve the expected performance gains @xcite . the key challenge is that the introduction of new position indices leads to an out-of-distribution (ood) issue @xcite . to alleviate the issues brought about by vanilla extrapolation, recent works have delved into the varying characteristics of positional encodings across different dimensional components. there has been a transition from linear interpolation @xcite methods to nonuniform interpolation and extrapolation methods @xcite . for instance, ntk-series methods @xcite @xcite address different dimensions of the positional encoding distinctly, while ntk-by-parts (bloc97, 2023) and yarn @xcite divide positional encodings into three groups based on frequency, each adopting different approaches. differentiating and processing positional encoding dimensions separately is an effective approach. notably, yarn has been adopted by several large language models @xcite , demonstrating its practical effectiveness in the community. a key distinction across dimensions lies in the periodic properties exhibited within the pretraining window. from this perspective, the difference between the high-dimensional (low rotational frequency) and low-dimensional (high rotational frequency) components in rope resides in their ability to learn complete and sufficient sine-cosine periodic patterns @xcite . extrapolating the low-dimensional components does not introduce new distributions in case of multiple complete sine-cosine cycles have already been learned, whereas the high-dimensional components behave in the opposite manner. therefore, current methods predominantly employ interpolation for the highdimensional components of positional encodings to mitigate severe ood issues caused by extrapolation @xcite @xcite . however, interpolation alters the rotational frequency, and learning new frequencies remains challenging, especially in the case of long extension length. in this paper, according to the distinct periodic characteristics of low-dimensional versus high-dimensional components in positional encodings, we introduce pepe (periodic extrapolation positional encodings), a new method for extending positional encodings. for high-dimensional components, pepe leverages a cyclic mechanism that reutilizes the incompletely learned periods, facilitating an infinite extension without altering the pre-trained rotational frequency distribution. in contrast, for low-dimensional components, we retain the conventional direct extrapolation approach. this strategy not only ensures efficient learning but also enhances the effectiveness of long-range extrapolation, addressing the ood challenge associated with extension context lengths. according to different cyclic modes, pepe can be implemented in two ways: periodic shift extrapolation (pse) and its mirrored approach (mpse). fig. 1 shows the comparison between the two modes of pepe and other extrapolation methods. pepe exhibits excellent stability in long-context expansion. we summarize the contribution as follows: ## periodic extrapolation positional encodings motivated by the periodic incompleteness of highdimensional components in positional encodings inevitably leads to ood issues, we present periodic extrapolation positional encodings (pepe), a positional encoding method that extends long-context sequences by periodically replicating pre-trained high-dimensional components. ## experiments to verify the effectiveness of our method, we first compared common extension methods on three tasks (sec. 4.2). then, we perform ablations of the hyperparameters that affect pepe's performance (sec. 4.3). additionally, we analyze two implementation modes of pepe (sec. 4.4). more details of experiments can be found in the appendix. ## conclusion this paper presents pepe, a novel positional encoding extension method designed to address the ood challenges. from a periodic perspective, by applying shift operations on high-dimensional components of positional encodings, pepe avoids introducing new positional information while maintaining compatibility with pre-trained models. based on pepe, we develop two practical variants, pse and mpse, both of which demonstrate strong stability under large extrapolation ratios. the results show that pepe achieves superior perplexity performance and retrieval accuracy compared to current methods. looking forward, pepe opens new possibilities for efficient context length extension in llms, with potential applications in long-text understanding, agent reasoning, and beyond.