Z ero SCROLLS : A Zero-Shot Benchmark for Long Text Understanding (2023)

Abstract: We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation of both open-source and closed large language models, finding that Claude outperforms ChatGPT, and that GPT-4 achieves the highest average score. However, there is still room for improvement on multiple open challenges in ZeroSCROLLS, such as aggregation tasks, where models struggle to pass the naive baseline. As the state of the art is a moving target, we invite researchers to evaluate their ideas on the live ZeroSCROLLS leaderboard.

## introduction large language models (llms) have been improving at an incredible pace, solving problems that seemed out of reach, without any task-specific training examples @xcite @xcite . as commercial llms are adopted worldwide, it becomes clear that they must also operate successfully over long sequences, such as conversation histories or scientific documents. however, current llm benchmarks that do evaluate models in a zero-shot setting, such as helm @xcite and bigbench @xcite , mostly focus on short sequences; bigbench, for example, has an average of 77 words per input. to fill this gap, we introduce zeroscrolls: zero-shot comparison over long language sequences, a benchmark for zero-shot long text reasoning over natural language, and conduct a thorough investigation of state-of-the-art llms. zeroscrolls extends scrolls @xcite , a long text understanding benchmark that 1 https://www.zero.scrolls-benchmark.com/ enables fine-tuning, adding four additional tasks: query-based summarization, multi-hop question answering, sentiment aggregation, and sorting book chapter summaries. we specifically design the latter two tasks to examine a model's ability to aggregate and compare information across long sequences, while keeping evaluation simple and accurate. zeroscrolls is designed to test zeroshot capabilities, and contains test sets with simple natural prompts and private gold references, small validation sets, and no train data. it has a live leaderboard to enable transparent and dynamic progress. figure 1 shows the state of the leaderboard based on our experiments, and figure 2 shows a per-task breakdown of a selected subset of models. we use this new testbed to perform extensive evaluation and analysis across state-of-the-art open and closed models. on question answering tasks, we find that zero-shot llms bridge the gap with task-specific fine-tuned models; gpt-4 sets a new state of the art on the challenging quality task @xcite , almost reaching human performance. in contrast, llms generally struggle to obtain such high scores for summarization tasks without a training set from which to learn the nuances and artifacts of each dataset, even though gpt-4 does approach the fine-tuned state of the art on two of three datasets. we also observe that two of our new tasks, sentiment aggregation and sorting book chapter summaries, prove exceptionally challenging for all llms, with only gpt-4 surpassing the naive baseline in each task. our code is available online. 2 when analyzing gpt-4 responses, we often find correct answers that do not match the requested format; e.g. producing a full sentence when asked to answer in a single phrase. this problem is not unique to gpt-4, as different models may deviate from the specified format in different tasks. while zeroscrolls is primarily aimed at facilitating research in understanding long texts, we encourage the community to use this benchmark to advance research in instruction understanding, prompt engineering, and evaluation of generated texts as well. all the aforementioned methods eventually finetune a specialized model for every single task in scrolls, a setting that remains important for many applications. however, in the modern era of general purpose, zero-shot reasoning llms, a new evaluation setup is required, where this dependence on task-specific fine-tuning is alleviated. ## the zeroscrolls benchmark zeroscrolls is a zero-shot benchmark containing test sets of ten natural language tasks, each one requiring reasoning over a different type of long text. to ensure affordability, we limit every task to a maximum of 500 examples. ## evaluating state-of-the-art llms using zeroscrolls we conduct, to the best of our knowledge, the first systematic llms zero-shot performance comparison over tasks that require long text understanding. ## analysis while gpt4 has the highest score on the zeroscrolls leaderboard, we find it surprising that other models score higher on a number of question answering tasks. we analyze model generations and observe that gpt-4 responses do not match the desired output format (despite explicit instructions in the prompt), which results in penalization by the automatic metrics. further analysis reveals that format discrepancy is a phenomenon that occurs across different llms and tasks, and is not unique to gpt-4 and question answering. ## conclusion we introduce zeroscrolls, a benchmark for zeroshot natural language understanding over long texts. zeroscrolls enables systematic comparison of llms on tasks with naturally long input texts, and ones that require contextualizing and aggregating information from multiple documents. we evaluate open-source and production-grade llms to find that gpt-4 and claude are currently the best performing models, while open-source models such as flan-ul2 also prove powerful at long-context question answering tasks. zeroscrolls remains an open challenge for llm research, with our two new aggregation tasks proving to be particularly difficult for contemporary llms.