Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles (2024)

Abstract: Previous research in multi-document news summarization has typically concentrated on collating information that all sources agree upon. However, the summarization of diverse information dispersed across multiple articles about an event remains underexplored. In this paper, we propose a new task of summarizing diverse information encountered in multiple news articles encompassing the same event. To facilitate this task, we outlined a data collection schema for identifying diverse information and curated a dataset named DiverseSumm. The dataset includes 245 news stories, with each story comprising 10 news articles and paired with a human-validated reference. Next, to enable consistent automatic evaluation, we conducted a comprehensive analysis to pinpoint the position and verbosity biases when utilizing Large Language Model (LLM)-based metrics for evaluating the coverage and faithfulness of summaries. Through correlation analyses, we outline the best practices for effectively using automatic LLM-based metrics on the DiverseSumm dataset. Finally, we study how LLMs summarize multiple news articles by analyzing which type of diverse information LLMs are capable of identifying. Our analyses suggest that despite the extraordinary capabilities of LLMs in single-document summarization, the proposed task remains a complex challenge for them mainly due to their limited coverage, with GPT-4 only able to cover under 40% of the diverse information on average.

## introduction in the realm of news reporting, each event is often chronicled by multiple sources, providing a rich tapestry of perspectives and insights. the sheer volume of articles available via news aggregators, as noted by @xcite , can overwhelm readers, leading to fatigue @xcite . this has fueled the demand for more digestible multi-source summaries. however, as highlighted by existing multi-document summarization studies @xcite @xcite , these often only reflect consensus information and neglect the breadth of differing viewpoints. to address this, we propose the multi-document diversity summarization (mdds) task, aimed at faithfully illuminating the diverse information presented in multiple sources. following @xcite , we formalize diverse information as questions and answers where numerous sources can answer the same question, and the corresponding answers extracted from different news articles exhibit a variety of opinions or perspectives. for robust and objective evaluation, we opted for a qa representation for references, aligning with the granularity and reliability advantages emphasized in prior work on summarization evaluation @xcite @xcite . an example of diverse information is shown in figure 1 . using this formulation, we propose a reference annotation methodology to identify and gather diverse information dispersed across multiple articles about the same story. our approach is a pipeline based on gpt-3.5-turbo (openai, 2023a), which generates questions concerning the story likely to pull varied responses from different sources. the subsequent answers extracted from each news article are then clustered into groups. we employ a post-processing step that removes invalid questions and answers. finally, all questions and answers are validated by human annotators. the resulting dataset contains 245 news story clusters, where each story contains 10 news articles and an average of 2.49 questions, with each question associated with 3.41 answer clusters on average. this dataset is named diversesumm. the articles present diverse and conflicting information regarding poland's stance on european integration, its support for ukraine, and its criticism of russia. while prime minister mateusz morawiecki warns against turning the european union into a "super-state government" that ignores national differences, he also emphasizes the importance of nation states in safeguarding the security and culture of european nations. poland's right-wing government is at odds with the eu on issues such as the rule of law and insists on the significance of individual member states' interests. additionally, morawiecki likens russian president vladimir putin to adolf hitler and argues that europe has a duty to oppose russian fascism. however, the polish ambassador's remarks about potential polish involvement in the conflict with russia were refuted by the polish embassy in france as being taken out of context. the embassy clarified that the ambassador was warning of the consequences of a ukrainian defeat rather than announcing direct involvement. overall, the articles highlight poland's support for ukraine, its concerns about european integration, and its criticism of russia, but also indicate varying views within the polish government and contradictory statements from polish officials. ## task the mdds task revolves around a cluster of k news articles all centered around the same news event. to maintain a balance between task feasibility and challenge, we have opted to set k at a value of 10. the primary aim of our task is to generate a natural-language summary that effectively captures the diverse information presented within this cluster of news articles. to facilitate this process, our data collection pipeline, as elaborated in §3, produces references for each news cluster. these references take the form of questionanswers (qas), and their validity is established through human validation. the qas must satisfy two properties: (1) the valid question must be answered by a sufficient number of sources, and (2) the answers associated with a valid question must present diverse opinions or perspectives. in this work, the assessment of the generated summaries centers on two key facets: faithfulness and coverage. the faithfulness aspect evaluates the extent to which the summary aligns with the factual content present in the source articles. on the other hand, the coverage aspect gauges the inclusivity of information by considering how many answers within the reference are effectively addressed in the summary. we set our primary focus on these two aspects instead of other qualities, such as compression ratio and coherence, because recent work has shown that faithfulness and coverage rq1: how proficient are llms in summarizing diverse information from multiple news articles about an event? rq2: what are the pitfalls and best practices when leveraging gpt-4 as the evaluation metric for our task? ## data collection this section details the diversesumm data collection pipeline, delineating its automated diverse information discovery from articles and the human validation stage that ensures data integrity. ## analysis we address the research questions from §1, first evaluating how well diverse information from multiple sources is summarized by llms ( §4.1), then examining llm behavior during this summarization ( §4.3) using the most reliable llm-based evaluation protocols we found ( §4.2). ## conclusion we introduce a novel task of multi-document diverse summarization that focuses on effectively summarizing diverse information from multiple news articles discussing the same news story. to facilitate this task, we construct a dataset, divers-esumm, using our proposed qa-based pipeline. through meticulous human evaluation, we have demonstrated that although llms exhibit a high level of faithfulness in tackling our task, achieving a high coverage rate remains particularly challenging, even with the most advanced llm like gpt-4. this underscores both the challenges and opportunities of mdds. furthermore, we have conducted an extensive analysis of bias and its correlation with human assessments across a range of evaluation protocols. leveraging the insights obtained from these experiments, we propose a set of recommendations that outline the most effective protocols for evaluating model performance within our task domain. our paper also delves into a comprehensive study that investigates llms' tendency to summarize various types of information. the outcomes of these analyses offer valuable insights into the behaviors exhibited by different llms when they engage with the challenge of summarizing diverse information. by presenting these resources and research findings, we hope to inspire and motivate future endeavors in the realm of comprehending and summarizing the intricate nuances present in diverse news articles concerning the same news event.