SCROLLS : Standardized C ompa R ison Over Long Language Sequences (2022)

Abstract: NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We examine existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input. SCROLLS contains summarization, question answering, and natural language inference tasks, covering multiple domains, including literature, science, business, and entertainment. Initial baselines, including Longformer Encoder-Decoder, indicate that there is ample room for improvement on SCROLLS. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods.

## introduction standard benchmarks à la glue @xcite , wmt @xcite , and squad @xcite , have driven progress in natural language processing of short utterances. however, a large portion of natural language is produced in the context of longer discourses, such as books, articles, meeting transcripts, etc. to tackle the computational challenges associated with processing such long sequences, a plethora of new model architectures have recently emerged @xcite , without establishing a standard scheme for evaluating them on long natural language problems. some long-context models are evaluated via language modeling perplexity, but this metric mostly captures model sensitivity to local, shortrange patterns @xcite ., 1 https://www.scrolls-benchmark.com @xcite and gpt3 @xcite ). scrolls incorporates multiple tasks (summarization, question answering, and natural language inference) over various domains (literature, meeting transcripts, tv shows, scientific articles, and more), where each example's input typically contains thousands of words. we review the existing literature on long-text tasks and manually curate a subset of 7 datasets, prioritizing those that require contextualizing and abstracting information across multiple parts of the text. we then clean and convert the data to a unified text-to-text format to enable the evaluation of a single model over all datasets. figure 1 shows that the texts in scrolls datasets are substantially longer than commonlyused nlp benchmarks. moreover, our analysis reveals that, in scrolls, critical information is spread out across longer distances within the input documents. scrolls is available via the datasets library @xcite or direct download on its website, which hosts a live leaderboard that accepts submissions and automatically evaluates them against private test sets. by producing a single aggregate score, in addition to individual dataset scores, scrolls can serve as an evaluation platform for future approaches to processing long text, whether by new pretraining schemes, novel transformer architectures and alternatives, or even retrieval-based methods. we provide initial baselines for scrolls using two transformer models, bart @xcite , and its length-efficient variant, longformer encoder-decoder @xcite . our experiments indicate that scrolls poses a formidable challenge for these models, leaving much room for the research community to improve upon. ## background: contemporary evaluation of long-text models while transformers @xcite are the current go-to architecture for building state-of-theart models in nlp, they present a computational challenge when it comes to long sequences due to the o(n 2 ) complexity of self-attention, where n is the sequence's length. to address this problem, a wide variety of efficient alternatives and approximations have been proposed over the past couple of years @xcite . much of these novel architectures were developed concurrently, leading to somewhat of a "wild west" when it comes to model evaluation, making crossmodel comparison challenging. roughly speaking, we can cluster the more prominent evaluation methodologies into three categories: language modeling, long-range arena, and summarization. the language modeling community typically uses perplexity to measure how well models predict the next token, a practice that has been adopted by several works on efficient transformer architectures @xcite @xcite . however, using perplexity to evaluate a model's long-range abilities is currently under scrutiny. a growing amount of literature shows that predicting the next token is mostly a local task that does not require modeling long-range dependencies @xcite , and that masking or down-weighting distant tokens can actually improve perplexity @xcite . a more recent approach to standardizing longsequence model evaluation is the long range arena (lra) @xcite . it incorporates 5 classification datasets: byte-level sentiment analysis (imdb) and document relatedness (acl anthology); path-finding (pathfinder) and image classification (cifar-10) over 1-dimensional pixel sequences; and executing a list of mathematical operations (listops). of those, two involve visual reasoning, and one is a synthetic mathematical language (listops), leaving only two natural language datasets (sentiment analysis and document relatedness). the multi-modal nature of lra makes it inappropriate as a testbed for pretrained language models, limiting its relevance for nlp. moreover, lra artificially inflates natural language sequences via byte tokenization, and truncates each example at 4,000 bytes, which is equivalent to less than 1,000 words. this exempts models from coping with the complex long-range dependencies that exist in naturally long texts. the third practice uses summarization tasks to evaluate long-sequence models. the most popular datasets use abstracts of academic papers on arxiv and pubmed @xcite as summaries. other summarization datasets, however, are less frequently used, biasing the evaluation towards academic domains. scrolls includes summarization as one of its main tasks, selecting datasets from several different domains to increase diversity. ## quantitative analysis length alone is not enough to make scrolls a challenging benchmark. here, we provide a quantitative analysis that suggests that producing the correct output for a scrolls task typically requires fusing different parts of the input that are often hundreds and even thousands of words apart. this analysis complements the qualitative inspection of examples from scrolls, as shown in figure 2 and figure 3 , and further discussed in appendix e. methodology each example in scrolls consists of a textual input and output. 15 given a specific input-output pair, we measure the example's spread by computing the standard deviation between the locations of output bigrams in the input. 16 specifically, we represent the output string as a set of bigrams, and locate the first occurrence of each bigram in the input (if exists); we then compute the standard deviation between these locations (where a bigram is represented by the position of its first word in the input). now that we have an example-level measure of spread, we can plot an entire dataset's spread on a histogram, and compare different datasets. summarization datasets figure 4a compares the three summarization datasets in scrolls to the canonical cnn/dm summarization dataset @xcite , as well as arxiv @xcite , which has been used to evaluate longsequence models. we observe that the reference bigrams are spread out across much larger distances in scrolls than in cnn/dm, and by a factor of 1.5 to 2 times more than arxiv on average. ## experiments we conduct experiments to evaluate the ability of mainstream models to handle the various long text challenges presented by scrolls. our code is based on the transformers library @xcite , and is available online. 13 ## conclusion we propose a new benchmark that places the spotlight on naturally long texts and their intricacies. scrolls fills a current gap around evaluating efficient transformer architectures and their alternatives on natural language tasks, and at the same time provides a testing ground for new pretraining schemes that target long language sequences. we hope that scrolls inspires the nlp community to go beyond single sentences and paragraphs, and meet the challenges of processing and reasoning over longer discourses.