C hapter B reak: A Challenge Dataset for Long-Range Language Models (2022)

Abstract: While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capabilities has not yet followed. To this end, we introduce ChapterBreak, a challenge dataset that provides an LRLM with a long segment from a narrative that ends at a chapter boundary and asks it to distinguish the beginning of the ground-truth next chapter from a set of negative segments sampled from the same narrative. A fine-grained human annotation reveals that our dataset contains many complex types of chapter transitions (e.g., parallel narratives, cliffhanger endings) that require processing global context to comprehend. Experiments on ChapterBreak show that existing LRLMs fail to effectively leverage long-range context, substantially underperforming a segment-level model trained directly for this task. We publicly release our ChapterBreak dataset to spur more principled future research into LRLMs.

## introduction research on long-range language models (lrlms) aims to process extremely long input sequences by making the base transformer architecture more efficient (e.g., through sparse attention, recurrence, or cached memory). these modifications are commonly validated by training lrlms on pg-19 @xcite , a long-document language modeling dataset, and demonstrating small perplexity decreases over shorter context models @xcite . however, recent analysis experiments @xcite show that modern lrlms rely mostly on local context (i.e., the immediately preceding 1-2k tokens) and are insensitive to various perturbations applied to more distant context. 1 we make our code and data public at https:// github.com/simengsun/chapterbreak in this paper, we move beyond token-level perplexity by evaluating lrlms on a task that requires a rich understanding of long-range dependencies. our task is an instance of suffix identification, in which a language model is given a long input sequence (or prefix) and asked to disambiguate the next n-token segment from a set of hard negatives sampled from the same narrative. to succeed at this task, an lrlm should assign high probability to the ground-truth next segment and low probability to the negatives. to specifically test long-range dependencies, we restrict our prefixes to end at chapter breaks of a longer cohesive narrative (e.g., a novel). we construct a challenge dataset, chapter-break, by automatically detecting chapter bound-aries within both held-out pg-19 documents (indomain for pretrained lrlms) and works of fan fiction published on the archive of our own (out of domain). 2 we perform a detailed analysis of the types of chapter transitions in our dataset and discover a high frequency of narrative shifts in pointof-view, location, and time, all of which require global narrative understanding over long input sequences. for example, figure 1 contains a complex prefix in which the time-traveling billy pilgrim moves between world war ii, 1960s suburban life, and an alien planet. understanding the cliffhanger ending, in which the narrative abruptly switches from a wartime scene to a 1967 alien abduction, requires an lrlm to make connective inferences using details buried far back in the context (e.g., billy's age in 1967). we evaluate three lrlms on chapterbreak, including bigbird @xcite , the routing transformer @xcite , and its local attention variant, all pretrained or fine-tuned on pg-19. our experiments show that these lrlms perform poorly at selecting the ground-truth suffix, regardless of the length of the input sequence. as an upper bound, we train a small roberta-based segment-level language model on pg-19 and discover that it substantially outperforms all lrlms on chapterbreak, which suggests that lrlms have considerable room for improvement on this suffix identification task. finally, we perform an analysis on the instances for which all models struggle to choose the correct suffix, which shows that shifts in location and events in focus are particularly challenging to disambiguate. taken together, these results suggest that chapterbreak is a useful benchmark for future research into lrlms. ## the chapterbreak dataset authors often break long-form narratives into a sequence of discrete chapters to impose "an order and shape over events in time" @xcite . henry fielding writes in his novel joseph andrews that the space between chapters is like "an inn or resting place" for readers to reflect on the preceding chapter @xcite . chapters come in many flavors: for example, murakami's kafka on the shore uses chapter breaks to alternate between parallel narratives focusing on the two protagonists, while cliffhanger endings such as the one in figure 1 add suspense. making sense of the complex narrative shifts associated with chapter transitions (e.g., changes in point-of-view, time, location, and theme) requires a deep understanding of the entire text. to maintain global narrative coherence, @xcite show that human readers tend to reactivate memory about "backgrounded" information from the long-range context. task overview: given that chapter transitions requires global context understanding, how can we turn this into a task to evaluate lrlms? a simple approach is to evaluate the token-level perplexity of an lrlm only at chapter boundaries (i.e., on the first n tokens of each chapter); however, the vast majority of tokens can be predicted using just local context @xcite under the teacherforcing setup, which obscures an lrlm's usage of long-range context as we show in section 3. we instead turn to the task of suffix identification, which closely resembles existing datasets such as swag @xcite . each instance of our task is defined by a triplet (c, s + , s - @xmath0, where c is a prefix sequence of up to 8k tokens that ends at a chapter break, s + is the gold suffix of length 128 tokens (i.e., the beginning of the next chapter), and s - i is a negative 128-token-long suffix from a set n of five foot_1 future chapter beginnings sampled from the same narrative. 4 all negatives are modified to begin with the same chapter index (e.g., if the gold suffix begins with "chapter iii", the chapter indices of all negatives is set to "chapter iii") to eliminate the effect found by @xcite of language models memorizing chapter indices in long contexts. we then evaluate whether an lrlm assigns higher probability to the gold suffix p (s + |c) than to all negative suffixes p (s - i |c). dataset overview: where do we get these triplets from? we collect a dataset, chapter-break, with two splits: chapterbreak p g19 , which contains 241 examples extracted from the pg-19 validation set @xcite ), foot_3 and chapterbreak ao3 , which contains 7,355 examples extracted from an online dump 6 of fanfiction posted on archive of our own (ao3). we apply filtering to remove fanfiction works that are too short or not rated for general audiences. each work contains on average 42k words and 21.5 chapters. 7 even though the chapterbreak p g19 split is small, we include it because many lrlms are pretrained on pg-19; the much larger chapterbreak ao3 split is outof-distribution for all models that we evaluate. to extract chapters in pg-19, we match for lines beginning with the string "chapter", while ao3 stories already have chapter-level metadata. what are the different types of transitions in chapterbreak and how often do they occur? to get a better sense of our dataset, we perform a fine-grained annotation of 300 randomly-selected chapter transitions from chapterbreak ao3 . for each transition, we annotate any changes in the following four aspects: events, actors (characters in focus), locations, and continuity. to annotate continuity, we follow a simplified version of the scheme proposed by @xcite , 8 which considers five categories: continuous (the next chapter occurs within a day of the previous chapter), discontinuous (the next chapter occurs more than a day after the previous chapter), analepsis (the next chapter is a "flashback" to an earlier point in the narrative), and parallel (the next chapter reverts to the time of a previous chapter, switching the 6 https://archive.org/download/ao3_ story_dump_continuing 7 more preprocessing details and statistics can be found in appendix a. 8 to validate our continuity annotations, we also annotate every chapter in pride and prejudice and obtain almost the same proportion of continuous transitions (67%) as the number reported by the expert annotation of ireland (1986) (72%). ## experiments we evaluate three different long-range language models on chapterbreak and compare their results to those of standard transformer language models as well as an upper bound directly trained for suffix prediction. language models: we evaluate three lrlms pretrained on pg-19: the local transformer @xcite , routing transformer (rt) @xcite , and bigbird @xcite . the bigbird model is the decoder part of the released checkpoint fine-tuned with causal lm objective on 14k books of pg-19 for 100k steps. we also evaluate two standard transformer language models, gpt-2 large @xcite and gpt-3 @xcite . 10 we summarize these models in an upper bound directly trained for suffix identification: as authors often write stories that are intended to surprise readers, it is possible that many examples in chapterbreak are ambiguous by nature (i.e., the upper bound for suffix identification accuracy may not be 100%). to obtain a reasonable upper bound, we also train a model (suffixlm) directly on the suffix identification task by scaling up the sentence-level language model proposed by @xcite . 11 we divide an input sequence into multiple segments, each of which is embedded via the [cls] vector of a small finetuned roberta network @xcite . our suffixlm then performs "language modeling" atop the dense [cls] vectors, predicting the next segment representation given the representations of previous segments via contrastive predictive coding (van den oord et al., 2018). 12 formally, our suffixlm minimizes the following loss: ## results & analysis overall, the results in accuracy increases with longer prefixes: figure 2 shows that as prefix sequence length increases, some lrlms (e.g., lt) barely improve, while others show modest improvements (e.g., gpt-3 and fine-tuned bigbird). however, all lrlms significantly underperform our suffixlm upper bound, even when the suffixlm is given prefixes that are only 256 tokens long. additionally, suffixlm's accuracy increases far more than those of lrlms when increasing the prefix length (from 31% at prefix length of 256 to 46% at 8k on the ao3 split foot_8 ). this result suggests that the tokenlevel lrlms evaluated in our work are not taking full advantage of information in the long-range context to solve chapterbreak. ## related work our work depends heavily on recent advances in efficient transformers @xcite ) that pro15 appendix a contains more details about data for these two discourse markers. 16 appendix g shows similar trends on cause and dialogue with other models. cess long sequences @xcite @xcite @xcite . sparse attention @xcite , relative position encoding @xcite @xcite , recurrence mechanism and memory @xcite @xcite and other tricks @xcite @xcite @xcite @xcite are commonly adopted by recent transformer variants to make the operation on long sequences more time/memory efficient. besides perplexity, many downstream extrinsic tasks for evaluating long-range language models were developed recently , such as long-form qa @xcite , documentlevel summarization @xcite , and machine translation @xcite . more recently, @xcite introduce a new benchmark covering multiple domains and tasks, while @xcite propose multimodal long sequence tasks. ## conclusion we introduce chapterbreak, a suffix identification dataset targeted at evaluating the discourselevel understanding of long-range language models. the dataset is extracted from long-form narratives and covers a variety of complex chapter transitions, such as shifts in location and events in focus. experiments show that existing lrlms perform poorly on chapterbreak and much worse than a suf-fixlm trained as an upper bound on this task. we release the dataset to spur more principled development of future lrlms. on archive of our own (ao3). we refer readers to @xcite for more details about pg-19. for ao3, we apply multiple filters to obtain long fanfiction stories rated as suitable for "general audiences". we refer readers to appendix a for more preprocessing details. more generally, this work focuses on long-range language models, which could potentially be misused to generate offensive output. however, the main purpose of this paper is to present a dataset which provides a better evaluation of the discourse-level capabilities of such models.