Do Long-Range Language Models Actually Use Long-Range Context? (2021)

Abstract: Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the Routing Transformer, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).

## introduction understanding long documents requires modeling various discourse-level phenomena, including anaphora (hobbs, 1979; grosz et al., 1995), argument structure (grimshaw, 1990), narrative scripts and trajectories (schank and abelson, 1977; labov and waletzky, 1997), and causal links between concepts (mooney and dejong, 1985). unfortunately, most language models (lms) are trained to predict the next word given only a small window of local context, which prevents them from using long-range discourse structure to improve their predictions. many research efforts over the years have attempted to address this issue: for example, rosenfeld (1996) incorporated statistics from distant tokens to improve n-gram models, while ji et al. (2015) added document-level context to neural lms. more recently, the transformer lm (vaswani et al., 2017), which forms the backbone of state-ofthe-art nlp systems (devlin et al., 2019; brown et al., 2020), has become the focus of numerous efforts to process longer input sequences. transformer lms are constrained by the inefficiency of the self-attention mechanism, whose complexity scales quadratically with the input sequence length. as such, more efficient methods based on sparse attention (correia et al., 2019) and cached memory (rae et al., 2020) have been proposed to increase the maximum sequence length, which has progressed from gpt-2's 1024 tokens (radford et al., 2019) to 4096 tokens (zaheer et al., 2020) and finally 8192 tokens (roy et al., 2021, routing transformer). when evaluated on the pg-19 benchmark dataset (rae et al., 2020), which contains long documents in the public domain, these "long-range" transformer lms also reach lower perplexities than baseline models on held-out data. how do "long-range" transformer lms make use of the long-range context? do they actually encode important discourse information to improve their predictions? in this paper, we conduct a series of fine-grained analysis experiments to answer these questions, several of which are inspired by the context analysis of lstm lms conducted by khandelwal et al. (2018). we focus specifically on analyzing the behavior of the state-of-the-art routing transformer and a simpler baseline model in the presence of various perturbations (e.g., word shuffling, random document replacement), and look closely at how different types of tokens are affected. our results show that: while modern lms can process much longer input sequences than those of the past, we conclude that they do not exploit the information available in the long-range context. we recommend that future research on long-range lms includes analysis experiments such as those in our work to shed light on how and when they are using the distant context. ## background & setup in this section, we first provide an overview of the long-range language models analyzed in this work (local transformer, routing transformer). next, we describe the experimental setup used in the remainder of the paper to measure the impact of the long-term context. ## the effect of longer context how does the size of the prefix affect the perplexity of long-range transformer lms? in this section, we evaluate our rt and lt checkpoints on the pg-19 validation set with varied prefix length. we discover that although these models are theoretically able to encode long sequences, increasing the prefix length beyond 2k tokens does not bring discernible improvements in aggregate. however, we do identify small subsets of tokens that benefit from long-range context. additionally, we find that these models take advantage of long-range context to different degrees on different types of books (e.g., continuous fictional narratives vs. discontinuous magazine articles). validation perplexity does not improve when the prefix length grows beyond 2k tokens: as shown in figure 1 , rt perplexity plateaus when evaluated with prefixes longer than 2k. @xcite in contrast, relative to rt, the perplexity curve for the more primitive lt starts flattening earlier at around 1k tokens (note that its effective context size is only 5k). we conclude that while rt's clustering heads take better advantage of global context than lt, the long-range context beyond 2k tokens is not helping overall. surprisingly, the perplexity gap between rt and lt is relatively consistent regardless of the prefix length, which indicates that much of rt's gains do not come from its increased ability to leverage long-range context but rather from better modeling of local context. infrequent tokens can benefit from increased prefix length: while the overall perplexity does not improve with increasing prefix length, we do observe different behavior when filtering the target tokens by frequency, as shown in figure 2 . we define frequent tokens to be the top 10% more frequently-occurring tokens in the subword vocabulary of pg-19 while the rest of tokens are clas- @xcite as a point of comparison, the perplexity of the much smaller lstm language models evaluated by khandelwal et al. (2018) plateaus after 200 words. additionally, press et al. (2020) discover that the perplexity flattens after 1k for a smaller standard transformer lm. 2500 5000 7500 prefix length 170 180 190 200 perplexity @xmath0: rt perplexity of target tokens whose last appearance is more than 2k tokens away in the prefix keeps decreasing. routing transformers are able to copy tokens that occur in the long-range context: target subword tokens that can be copied from somewhere in the prefix form another interesting group to analyze. @xcite while this is commonplace for frequent words (e.g., determiners, pronouns), it also occurs for entities and rare words (e.g., character names in a novel); sometimes, a word can occur several thousand tokens after its last occurrence. we focus on the latter category of tokens, specifically using a prefix length of 2k tokens as a cutoff to distinguish local and long-range context. perplexities are computed over 22k tokens which occur last time more than 2k tokens away, and 36k tokens that never appear in the prefix. in particular, the left plot in figure 4 shows the perplexity of tokens that cannot be found in the previous 2k tokens, but occur somewhere in the long-range context (2k to 8k tokens away). while the lt curve for such tokens plateaus after 2k tokens, indicating that lt cannot take advantage of repeated words in the distant following patterns in the long-range context: besides the token categories examined above, we also qualitatively look at some examples that are too infrequent to analyze at scale. interestingly, we observe some simple patterns (slightly more complex than copying) that the rt model picks up on. specifically, it learns to increment chapter numbers even if the previous chapter title appears more than 2k tokens away: for example, when predicting "chapter v" in the validation book keith of the border, modifying the previous chapter title "chapter iv", which occurs 2300 tokens away, to "chapter v" causes the loss of the predicted token "v" to increase by over 10. the impact of book type on the benefits of longrange context: pg-19 contains a diverse array of topics, genres, and formats, not all of which equally benefit from long-range context modeling. for example, while continuous narratives (e.g., novels) certainly build up many high-level discourse structures over a long sequence of tokens, discontinuous text like magazines, textbooks, or short story collections may require primarily local modeling. to better understand the effect of the type of book on long-range lm perplexity, we annotate every book in pg-19's validation set as either fiction or nonfiction and continuous 13 or discontinuous. @xcite out of 49 books we annotated, 30 are non-fiction, 15 31 are discontinuous, and 25 books are both non-fiction and discontinuous. we observe in figure 5 that the rt model takes better advantage of long-range context for fictional and continuous books, as the perplexity for these books plateaus at around 5k tokens. figure 6 shows fictional and continuous books exploit better the long-range context while predicting tokens within subword clusters. overall, we find the improvement stems largely from continuous and fictional books; more details are included in appendix b. ## the perturbation of long-range context the experiments in the previous section show that incorporating long-range context (further than 2k tokens away from the target) yields only marginal improvements to the overall perplexities of rt and lt. however, the long-range context does have a notable positive impact on a subset of tokens and book types. do these improvements persist in the presence of severe perturbations to the distant context? if so, this would indicate that they are not encoding any complex discourse structure in the context but rather relying on surface information (e.g., token presence) to make better predictions. in @xcite we consider books with related but distinct sections (such as textbooks) to be discontinuous in our annotation. @xcite we also annotate whether the work has been written by the same author or various authors, which is presented in the appendix. we define the following three perturbation operations for ρ and report results averaged over five runs for each of them. tokens inside subword clusters and tokens repeated in the distant context depend on word order: similar to the analysis in section 3, the experiments above may hide impacts on small subsets of tokens, which motivates us to do a more fine-grained analysis. we find that tokens inside subword clusters (figure 10 ) and those that can only be copied from long-range context (figure 11 ) are sensitive to both the order and the content of the remote context. while random replacement is more harmful than shuffling for tokens that can be copied in the distant context (172 shuffled vs 174 random replacement perplexity when perturbing 6k tokens), shuffling is more detrimental for tokens inside subword clusters (3.8 vs 3.7 perplexity when perturbing 6k tokens). routing transformer encodes token identity in the long-range context: while the previous perturbations affected entire contiguous blocks of the prefix, we move now to more targeted perturbations. an interesting question to ask given the observation that rt perplexity decreases on copied tokens as sequence length increases ( § 3) is how much that perplexity decrease depends on word order and surrounding content. in response, we drop tokens in the distant context whose next appearance is in the target sequence. as a control experiment, we drop the same number of random tokens for each perturbation length. as shown in the right plot of figure 12 , dropping the previous long-range occurrences of target tokens increases the perplexity of those target tokens, which shows that rt indeed memorizes token identity in the long-range context to some extent. the left plot shows that dropping long-range duplicate tokens does not affect tokens that also occur within the local context (i.e., the prior 2k tokens). the flat curve before 6k indicates the model relies only on the most recent occurrences for prediction. ## sequence-level analysis all of the previous experiments have focused on token-level perplexity, which is the standard way in which lms are evaluated. however, the prefixes in these evaluations consist solely of ground-truth text, mirroring the "teacher-forcing" setup that lms are trained with. when these models are deployed practically to generate text, they have to rely on their previous predictions instead of ground-truth text, and several prior works have noted different behavior in this setting (wang and sennrich, 2020; holtzman et al., 2020; welleck et al., 2020). in this section, we shift from token-level tasks to analyzing rt and lt performance on sequence-level tasks. in particular, we first look at how well the models can memorize an exact sequence in the distant context, as opposed to a single token as we did previously. next, we examine the models' ability to identify which of six 128-token suffixes follows a given prefix, which examines their behavior outside the standard teacher-forced setting. sequence-level copying: as a sequence-level analogue to the token-copying analysis in the previous section, we examine both rt and lt's ability to memorize a sequence that occurs in the distant context. to test this ability, we copy the target sequence and paste it into different positions of the prefix. the left plot in figure 13 shows that both models give a very low perplexity to the target sequence if its duplicate appears within the previous 512 tokens. however, both models lose their ability to take advantage of the copied sequence if it occurs more than 2k tokens away. this confirms our previous discovery that sequence order is in general not encoded in the long-range context. ## related work our work examines recent advances in efficient transformer variants (sukhbaatar et al., 2019; kitaev et al., 2020; choromanski et al., 2021; tay et al., 2021; katharopoulos et al., 2020; wang et al., 2020; wu et al., 2020) that accept longer sequences than prior approaches. longer effective context size is often achieved by sparse attention (child et al., 2019), recurrence (dai et al., 2019), and cached memory (weston et al., 2015; rae et al., 2020). our work is also related to methods that incorporate long context (wang and cho, 2016) as well as document-level tasks that inherently require modeling long-range context (zhang et al., 2018; hofstätter et al., 2020; zhang et al., 2020). this work is also similar to other analysis of language models, especially for long-range context. khandelwal et al. (2018) analyze the usage of long-term context of smaller lstm lms. sharan et al. (2018) prove long-term context is not needed for hmm lm due to teacher forcing. rae and razavi (2020) conduct an analysis exclusively for the transformer-xl (dai et al., 2019) model. rae et al. (2020) show that compressive transformer improves the performance of infrequent tokens. our work also relates to that of lai et al. (2020), who investigate the impact of context for pretrained masked lms. more recently, press et al. (2020) also observe negligible benefits of long-term context; we step further in this direction by exploring larger models with more fine-grained analysis. ## conclusion we perform a fine-grained analysis of the impact of long-range context to both token-and sequencelevel improvements on two long-range transformer language models, using the pg-19 dataset as a testbed. our results suggest these models rarely take advantage of the long-term context, and when they do it is mostly in superficial ways (e.g, by copying rare tokens from far away). with the proliferation of research in increasing the input size of transformer lms, we hope that our research will lead to more meaningful progress on integrating discourse information into these models.