Length Generalization of Causal Transformers without Position Encoding (2024)

Abstract: Generalizing to longer sentences is important for recent Transformer-based language models. Besides algorithms manipulating explicit position features, the success of Transformers without position encodings (NoPE) provides a new way to overcome the challenge. In this paper, we study the length generalization property of NoPE. We find that although NoPE can extend to longer sequences than the commonly used explicit position encodings, it still has a limited context length. We identify a connection between the failure of NoPE’s generalization and the distraction of attention distributions. We propose a parameter-efficient tuning for searching attention heads’ best temperature hyper-parameters, which substantially expands NoPE’s context size. Experiments on long sequence language modeling, the synthetic passkey retrieval task and real-world long context tasks show that NoPE can achieve competitive performances with state-of-the-art length generalization algorithms. The source code is publicly accessible

## introduction causal transformer has been widely applied in modern language models. to help models recognize the correct ordering of words, it is common to configure transformers with explicit position encodings (e.g., the sinusoidal embeddings in the original development of transformer @xcite , the relative position encoding in t5 @xcite , and the rotary position encoding in gpt series @xcite ). the setup of position features provides flexibility to include prior knowledge structure on describing distance, but it also brings the problem of length generalization: language models trained with in-domain position features can not handle longer sentences (i.e., those with out-of-domain position features) in testing time. generalizing to unseen sentence length is crucial in many language model applications like retrieval augmented language models @xcite , personalized language models @xcite , language-model-based agents @xcite . departing from the standard ways of encoding positions, one may ask (following the principle of parsimony) that are the explicit position features necessary? the answer is no. both empirically @xcite and theoretically @xcite , the casually masked transformers are shown to be able to successfully model languages without any prior position encoding (nope). the finding calls for a deeper understanding of implicit position information in transformer-based language models, and also inspires a new direction for length generalization: without explicit position features, can nope generalize? in this paper, we study the length generalization property of nope. our main findings are, we conduct length generalization experiments on long sequence language modeling, synthetic tasks (passkey retrieval), and longbench. the results show that nope enjoys a competitive extension performances to state-of-the-art length gener-alization methods for explicit position encodings (e.g., pi @xcite , yarn @xcite ). ## head-based attention scale after verifying that the attention scaling can help nope generalizing, we delved deeper into the multi-head attention mechanism and posed a new question, "does each attention head require a unique scaling factor?" in this section, we first visualize the average entropy curves for each head and find that they have different attention patterns. hence we propose to replace the uniform scaling with head-based scaling (from one factor to 22 × 32 = 704 factors). to address the issue of an exploding search space, we efficiently determine the values of scaling factors through automated hyperparameter search, considering both parameter efficiency and data efficiency. as a result, head-based scaling generalizes better than uniform scaling. moreover, correlation analysis shows that within each layer, the smaller the converged entropy (i.e., the more concentrated attention), the larger the required scaling factor to maintain that concentration. ## experiment we train a nope base model from scratch and investigate its capability in length generalization. we conduct length generalization experiments on long sequence language modeling, synthetic tasks (passkey retrieval), and real-world long context tasks (longbench). detailed experiment setup can be found in appendix a. ## discussion we studied the length generalization of casual transformer without explicit position encoding. we developed a parameter-efficient tuning algorithm which aims to search for the best temperature hyper-parameters for attention heads. through empirical evaluation, we saw that nope can achieve competitive length generalization and might be a promising alternative for long-context language modeling. nope provides a new perspective to understanding the role of positional information by isolating and eliminating the effects of explicit positional encoding. our work demonstrates the correlation between length generation failures and distraction of attention in nope models, thus the proposed method concentrates the attention by adjusting the scaling factor. while current works on length generalization mainly focus on manipulating positional encoding, our work suggests a new key component to generalization.