Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches (2025)

Abstract: Automatically summarizing large text collections is a valuable tool for document research, with applications in journalism, academic research, legal work, and many other fields. In this work, we contrast two classes of systems for large-scale multi-document summarization (MDS): compression and full-text. Compression-based methods use a multi-stage pipeline and often lead to lossy summaries. Full-text methods promise a lossless summary by relying on recent advances in long-context reasoning. To understand their utility on large-scale MDS, we evaluated them on three datasets, each containing approximately one hundred documents per summary. Our experiments cover a diverse set of long-context transformers (Llama-3.1, Command-R, Jamba-1.5-Mini) and compression methods (retrieval-augmented, hierarchical, incremental). Overall, we find that full-text and retrieval methods perform the best in most settings. With further analysis into the salient information retention patterns, we show that compression-based methods show strong promise at intermediate stages, even outperforming full-context. However, they suffer information loss due to their multi-stage pipeline and lack of global context. Our results highlight the need to develop hybrid approaches that combine compression and full-text approaches for optimal performance on large-scale multi-document summarization.

## introduction summarizing events described in document collections has long interested the nlp community with shared tasks for event tracking @xcite and summarization @xcite @xcite . given an input collection of hundreds of text documents, systems have to extract and summarize salient information about the event. the length and diversity of the input presents a challenge to recent large language models (llms). in this work, we contrast two classes of systems for large-scale multidocument summarization (mds), compressionbased, and full-text systems. 2 full-text systems promise a lossless approach by providing the summarizer access to the entire input. they are based on the long-context reasoning abilities of lms, having already shown strong retrieval performance on long inputs @xcite . however, their capabilities on large-scale mds are not as well understood. in a recent work, @xcite introduced a synthetic mds benchmark that resembles the needle in a haystack evaluation @xcite . in addition to this dataset, we evaluate on two large-scale event summarization datasets: background @xcite and wcep @xcite . we contrast the end-to-end full-context method foot_2 with three compression-based methods: retrieval, hierarchical, and incremental. each method compresses the input in a multistage pipeline ( §2.2). we evaluated the content selection aspects of the summary using the atomic content unit (a3cu) metric @xcite . our experiments show that full-context and retrieval perform best in most settings ( §3). to better understand the performance of compressionbased methods, we measure a3cu recall to track the salient information retention in their intermediate outputs ( §3.4). across all settings, we find that compression-based methods show high recall in intermediate stages but suffer information loss in their multistage pipeline. in particular, the intermediate recall is often much higher than the full-context system recall. we highlight two key takeaways: first, while iterative methods (hierarchical & incremental) were previously found effec-tive for book summarization and small-scale mds, they underperform on large-scale mds. second, full-context systems are suboptimal on large-scale mds datasets. we advocate for hybrid methods that combine input compression and long-context models. such hybrid approaches are also scalable to even larger mds tasks that go far beyond the context window limits of current llms. ## conclusion in this work, we contrast the full-context method against three compression-based methods for largescale mds. we evaluated on three datasets, summhay, background, and wcep using the a3cu content selection evaluation metric. we find that the full-context and retrieval-based methods perform the best. iterative methods suffer from significant information loss. our analysis shows that full-context methods provide suboptimal performance, and we recommend future work to explore hybrid methods that combine the strengths of input compression methods with advances in long-context llms.