Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis (2023)

Abstract: Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences.A relative positional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empowered by a novel cumulative normalized gradient tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal positional embedding to create Sandwich , the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarithmic decaying temporal bias pattern with learnable relative positional embeddings; these elucidate future extrapolatable positional embedding design.

## introduction the length of input sequences is an important hyperparameter choice for pretraining a transformer language model. a vanilla transformer language model has a quadratic training cost w.r.t l tr , the training sequence length. as the value of l tr increases, cost becomes impractical. however, we can use the model for substantially longer evaluation sequence lengths l ex l tr as gradients no longer need to be recorded. the discrepancy between l tr and l ex motivates the task of length extrapolation @xcite : can a transformer language model maintain equally good, if not better, perplexities when longer sequences are used in the testing stage? several extrapolatable transformer language models have been proposed including alibi @xcite and kerple @xcite , of which the relative positional embedding design is hypothesized to be critical to success. empirically, they extrapolate to l ex l tr much better than other absolute and relative positional embeddings for a transformer language model with h attention heads, the range of h is n • 8 h , where @xmath0. . . h}. @xmath1, @xmath2. figure 2 : windowed attention. this is the same design as longformer @xcite . we limit the context window size to @xmath3. @xmath4, @xmath5. windowed attention allows the easy derivation of a theoretical (maximum) receptive field: wr for an r layer transformer model with windowed attention size w. a windowed attention model can extrapolate if l @xmath6, and 2) it simply ignores the additional l ex -wr tokens during the testing stage. surprisingly, a model can still extrapolate when l @xmath7. this calls for the need for empirical receptive field measurement and motivates our model-agnostic cumulative normalized gradient tool. the tool we develop can be applied back on alibi to show that l tr covers most of its empirical receptive field. our analysis tool also provides critical context for explaining the length extrapolation failure @xcite of sinusoidal @xcite and rotary @xcite by showing their violation of the empirical receptive field coverage principle. sinusoidal can be fixed by dropping the intermediate terms and keeping only the decay-with-distance biases; this leads to the creation of sandwich, the first parameter-free relative positional embedding that uses information beyond l tr . sandwich shares a similar temporal bias pattern with trainable positional embeddings such as kerple @xcite and t5 @xcite , and they jointly suggest the future design of extrapolatable transformer positional embeddings. ## alibi and windowed attention here, we alter the slope ( 1 2 h ) of alibi to check if the length extrapolation property persists and reveal the connection between alibi and windowed attention. we present three experiments on two datasets, arxiv and openwebtext2 (appendix a), to ensure that the observations are consistent across different text domains, shown in ## receptive field measurement following the definition of windowed attention size w, an r layer transformer has a theoretical receptive field (trf) of wr, which is the maximum number of tokens that contribute to the prediction of the next token. in practice, a neural model often uses a subset of trf, named empirical receptive field (erf). while previous work @xcite @xcite @xcite aims to increase erf to match trf, we show that decreasing erf could serve as one feasible approach to enable successful length extrapolation. consider the case where @xmath8: this model can extrapolate easily because its trf is fully covered and trained. concretely, if we set @xmath9, l @xmath10, we know that as long as @xmath11.6 = 512/12, trf will be fully covered by l tr . surprisingly, the model is still able to extrapolate up to @xmath12, leading to a trf of 100 * 12 = 1200 512. this can be explained by the erf and trf discrepancy discussed above; this calls for the need to quantify erf. ## conclusion in this paper, we first establish the connection between alibi and windowed attention through their constructions and language modeling performance. we then develop a cumulative normalized gradient tool to measure the empirical receptive field. it shows that length extrapolation of alibi and windowed attention is possible when the training sequence length covers the empirical receptive field. it also reveals the models' limitation of not utilizing information beyond the training sequence length. fortunately, this is overcome by our new relative positional embedding, sandwich, which is simplified from the earliest proposed sinusoidal positional embedding. finally, sandwich demonstrates a log-decaying temporal bias pattern similar to that previously seen in the design of kerple and t5, and such pattern is likely to be the secret to successful length extrapolation. together these findings supports more effective design of future extrapolatable transformer language models.