Summary of a Haystack: A Challenge to Long-Context LLM s and RAG Systems (2024)

Abstract: LLMs and RAG systems are now capable of handling millions of input tokens or more. However, evaluating the output quality of such systems on long-context tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity. In this work, we argue that summarization can play a central role in such evaluation. We design a procedure to synthesize Haystacks of documents, ensuring that specific insights repeat across documents. The “Summary of a Haystack” (SummHay) task then requires a system to process the Haystack and generate, given a query, a summary that identifies the relevant insights and precisely cites the source documents. Since we have precise knowledge of what insights should appear in a haystack summary and what documents should be cited, we implement a highly reproducible automatic evaluation that can score summaries on two aspects – Coverage and Citation. We generate Haystacks in two domains (conversation, news), and perform a large-scale evaluation of 10 LLMs and corresponding 50 RAG systems. Our findings indicate that SummHay is an open challenge for current systems, as even systems provided with an Oracle signal of document relevance lag our estimate of human performance (56%) by 10+ points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to study enterprise RAG systems and position bias in long-context models. We hope future systems can equal and surpass human performance on SummHay.

## introduction recent progress in efficient attention mechanisms has led to the expansion of the context length of large language models @xcite . previous state-of-the-art models such as t5 @xcite and bart (lewis et al., * equal contribution 2019) were limited to input contexts of 512 or 1024 tokens, while the latest models such as claude-3 or gemini-1.5-pro @xcite can process sequences of hundreds of thousands or millions of tokens. another paradigm, retrieval augmented generation (rag) @xcite , has emerged as an alternative to these long-context llms, proposing a pipelined approach in which a retriever dynamically selects the context relevant to a given input query, alleviating the need for the generator to process long contexts directly. although both rag and long-context llms offer to solve the common problem of answering queries over a large corpus of text, a direct comparison on a common task is still lacking, and evaluating such systems is an open challenge. some recent work has popularized tests such as the needle-ina-haystack task @xcite , which requires models to identify a small piece of information in a large document. however, these tasks do not offer the complexity needed to differentiate the capabilities of the latest generation of large-language models, with several state-of-the-art models achieving near-perfect performance. in this work, we propose to leverage the task of summarization as a testbed for evaluating longcontext models and rag systems. summarization requires reasoning over a long context and a careful understanding of the relative importance of content. however, most prior work on summarization evaluation, particularly in evaluating the relevance of summaries, has focused on single-document summarization or tasks in which the input content is on the order of 1,000-2,000 tokens @xcite @xcite . apart from @xcite , which focuses on summary coherence for 100k-token books, other evaluation work on longer conversational and multi-document news summarization is still often limited to around 10k tokens @xcite . a central problem is that summarization evaluation often relies on low-quality reference summaries and automatic metrics that do not correlate well with human judgments. within referencebased evaluation, a candidate summary is compared to a gold-standard reference summary, with the optics that a higher overlap between the candidate and reference summary indicates higher quality. this paradigm may limit evaluation reliability, due to the lack of gold-standard references, particularly in long-context settings where obtaining high-quality summaries would be prohibitively expensive. furthermore, automatic metrics may still fail to correlate well with human judgments with respect to these references; despite the humanvalidated pipeline of @xcite , the best automatic metric for content coverage in that study has a correlation of just 0.37 with human judgment. in this work, we address these limitations through synthetic data generation. an overview of our framework is found in figure 1 . we propose data synthesis programs to generate a large corpus of documents (the "haystack") on a given topic. by enforcing that certain units of information ("insights"), categorized according to various subtopics, repeat within haystack documents, and precisely controlling which insights occur in which documents, we can automatically derive the relevant insights within the haystack for a given search query. a system completing the summary of a haystack (summhay) task must then summarize insights relevant to a search query and cite the source documents of each insight. these summaries can be evaluated based on whether they cover the expected reference insights, and cite precisely and thoroughly the source documents. our first contribution is a procedure for generating haystacks in two domains: conversations and news articles. section 3 details the carefully designed pipeline to ensure the feasibility and validity of the task. a haystack typically contains 100 documents on a topic, totaling approximately 100k tokens. we generate a total of 10 haystacks, each coupled with roughly 10 queries, for a total of 92 summhay tasks. our pipeline can be scaled and applied to other domains. our second contribution develops summhay's evaluation protocol, centering on evaluating system outputs on their coverage of reference insights, and the quality of their citation. a manual annotation confirms the strong reproducibility of the protocol among knowledgeable annotators (0.77 correlation). we then experiment with llm-based evaluation, finding that although the level of correlation is slightly lower (0.71), evaluation cost is reduced by a factor of almost 50. our third contribution is an estimate of human performance on summhay and a large-scale evaluation of 50 rag systems and 14 long-context llms. our findings indicate that: (1) summhay is challenging for all systems we evaluate, with all models significantly below our estimate of human performance, even when given oracle signals of document relevance; (2) non-trivial trade-offs exist when choosing between a rag pipeline and a longcontext llm, with rag systems typically improving citation quality, at the cost of insight coverage, (3) using advanced rag components (e.g., cohere's rerank3) leads to end-to-end performance boosts on the task, confirming that summhay is a viable option for holistic rag evaluation, (4) a positional bias experiment on summhay confirms the lost in the middle phenomenon, demonstrating that most llms are biased towards information at the top or bottom of the context window. our code and dataset are publicly released to allow for the extension of the work foot_0 . a system that achieves a high score on summhay can reliably reason over large corpora of documents, detect and summarize insights, and accurately cite its sources. although human performance on summhay is still out of reach, we anticipate future systems will achieve and surpass such performance, providing more reliable and trustworthy answer engines. ## summary in a haystack framework figure 1 illustrates the process of synthesizing haystack data, and the task, which we now detail.