The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval (2025)

Abstract: The Rotary Position Embedding (RoPE) is widely used in the attention heads of many large language models (LLM). It rotates dimensions in the query and the key vectors by different angles according to their positions in the input sequence. For long context modeling, the range of positions may vary a lot, and thus RoPE rotates some dimensions by a great range of angles. We hypothesize that the wide range of rotation angles may prevent LLMs from utilizing those dimensions. To validate this hypothesis, we present a controlled experiment showing that applying RoPE causes low utility of certain dimensions. Our analyses on three LLMs also indicate that these dimensions do not help LLMs do long-context question answering.

## dimension inefficiency we hypothesize that rope may cause dimension inefficiency in attention heads for long-dependency modeling. specifically, when a task requires an attention head to attend to a distant position, rope may prevent the attention from utilizing the first few dimensions in its attention heads. this is because rope rotates those dimensions with greater rates (θ's in eq. 1) 1 . for long-context tasks, such as long-context question answering, the possible relative positions m-n between a key vector k n for the target information and a query vector q m can vary greatly, so the rotation applied on these dimensions can be any angles. therefore, the model cannot produce query and key vectors such that their first few dimensions can consistently contribute a positive value to the query-key inner product. we hypothesize that the first few dimensions are thus useless for long-distance attention. ## controlled experiment we present a controlled experiment to demonstrate how rope can cause dimension inefficiency. we design a simple experiment where the model needs to learn n vector tuples {(q i , k i , v i )} n @xmath0 in practice, the dimensions are reordered for computational efficiency. here we assume that the order of the dimensions is by the magnitude of θ, from greater to smaller. figure 2 : the average importance of each dimensions in the query vectors of the attention heads, measured by the l1 norm of rows in the query weight matrices (left) and by utility score in §5.2 (right). we visualize all heads in figure 5 and figure 6 . we train models in two setups, one with rope applied on k and the other without (details in §a). results and discussion our experimental results indicate that rope causes dimension inefficiency. firstly, we plot the average weight of {q} n @xmath1. it shows that the model trained with rope applied learns to assign lower weights to the first few dimensions of {q i } n @xmath2. secondly, figure 1b shows that, when rope is applied, removing the first few dimensions does not affect the loss significantly, while removing the last few dimensions greatly increases the loss. this indicates that the model relies mainly on the last few dimensions and does not utilize the first few dimensions. in contrast, the models without rope do not exhibit these phenomena. this is in line with our hypothesis in §3. ## inspecting real-world models we then inspect three 7b/8b large language models (llm), llama-3.1-8b-instruct @xcite , qwen-2.5-7b-instruct @xcite , and olmo-2-7b-instruct @xcite . these models have 128 dimensions in their attention heads. for quick inspection, we first plot the l1 norm of the rows in the query projection matrices in all the attention heads in figure 2 . it shows increasing trends for all three models, as we have found in llama olmo qwen original 54.09 56.66 58.72 masked 54.15 57.55 57.36 the toy experiment (figure 1a ). as the l1 norm of the rows may not directly reflect the importance of the dimensions in the query vectors, we utilize a long-context task for further investigations. ## conclusion in this work, we hypothesize that the rotary position embedding (rope) may prevent lms from utilizing all the dimension for long-context modeling. we also provide supporting evidence, including a toy experiment §4, and a deep inspection of three llms §5. based on our finding, we suggest that future llm creators consider alternatives of rope, or at least not use rope for all attention heads.