One Thousand and One Pairs: A “novel” challenge for long-context language models (2024)

Abstract: Synthetic long-context LLM benchmarks (e.g., “needle-in-the-haystack”) test only surface-level retrieval capabilities; but how well can long-context LLMs retrieve, synthesize, and reason over information across book-length inputs? We address this question by creating NoCha, a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books, written by human readers of those books. In contrast to existing long-context benchmarks, our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. Our experiments show that while human readers easily perform this task, it is enormously challenging for all ten long-context LLMs that we evaluate: no open-weight model performs above random chance (despite their strong performance on synthetic benchmarks), while GPT-4o achieves the highest pair accuracy at 55.8%. Further analysis reveals that (1) on average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning; (2) model-generated explanations for their decisions are often inaccurate even for correctly-labeled claims; and (3) models perform substantially worse on speculative fiction books that contain extensive world-building. The methodology proposed in NoCha allows for the evolution of the benchmark dataset and the easy analysis of future models.

## introduction the context size of large language models has increased by multiple orders of magnitude over the last year: for instance, google's gemini pro 1.5 can process millions of input tokens at once. but can models truly utilize and reason over their claimed context? existing long-context evaluation methods such as finding the "needle in the haystack" (niah) (kamradt, 2023) measure surface-level retrieval capabilities, but do not necessarily assess performance on the more challenging task of synthesizing distant and underlying information as we show in §4. we bridge this gap by introducing nocha (a novel challenge), in which llms are prompted to verify claims written about recently-published fiction books. claim verification has been extensively studied in the context of shorter documents (thorne et al., 2018; wadden et al., 2020; fabbri et al., 2022a), but its application to book-length fictional texts presents unique challenges. firstly, the task necessitates reasoning over both explicit information directly stated in the text and implicit information inferred from the narrative, which is often distributed throughout the entire document. secondly, the use of recent, fictional context prevents the model from relying solely on parametric knowledge, necessitating the comprehension and interpretation of the long text. our data collection process aims to balance efficiency and quality. rather than pre-selecting a set of books, we follow the approach of kim et al. (2024) and ask annotators to self-report recently published novels they have read. the annotators then create true/false narrative minimal pairs that isolate a single narrative phenomenon present in their novels. each false claim differs from the true claim in its pair only by the inclusion of false information regarding the same event or entity (see figure 1 ). this approach offers two key advantages: (1) it minimizes the chances that the model is "correct for the wrong reason," as it must accurately predict both labels in the pair, and (2) it simplifies the process of quality control, as the false claim can be easily verified against its true counterpart, making it easier to identify claims that are either too similar or overly subjective. nocha contains 1,001 narrative minimal pairs for 67 books, created at a total cost of @xmath08k usd. since we found that open-weight models struggle to follow the prompt's instructions, we also experiment a simplified prompt that just asked for a true or false decision (see ## results & analysis model ruler (%) vanilla niah ruler (%) niah suite nocha (%) gpt-4-turbo 100.0 89.6 40.2 command r 98.0 84.8 19.6 / 22.5 @xmath2. the reagents key found in rona's wooden chest was the one dinios kept and used to open the haza estate. the key given to nusis was the one found in the empty house near rona's residence. </@xmath3. she runs it under tests, but they are inconclusive. </@xmath4 by using her skills as an apoth, nusis is able to reverse engineer the type of portal opened by the reagents key found in rona's wooden chest. ## conclusion and discussion we introduce nocha, a claim verification dataset designed to evaluate long context llms in a realistic task setting. our design ensures that most claims necessitate global reasoning over extended contexts for verification and cannot be easily "gamed" by relying solely on parametric knowledge or generating correct answers through incorrect reasoning. our experiments with 11 different long-context llms (5 open-weight, 6 closed-source) demonstrate that all models struggle on nocha. furthermore, we reveal a substantial competency gap with human readers who can very easily perform this task. importantly, our results show that models that are "state-of-the-art" according to synthetic benchmarks like niah actually perform very poorly on our meticulously designed dataset. nevertheless, we argue that complex synthetic datasets (such as ruler) are useful and complementary to our realistic dataset; they allow for much higher flexibility such as easily adjusting for different context lengths or analyzing the lost-in-the-middle phenomenon. we encourage researchers to use a holistic approach and consider both synthetic and realistic tasks when evaluating long-context language models.