An Empirical Analysis of Diversity in Argument Summarization (2024)

Abstract: Presenting high-level arguments is a crucial task for fostering participation in online societal discussions. Current argument summarization approaches miss an important facet of this task—capturing diversity —which is important for accommodating multiple perspectives. We introduce three aspects of diversity: those of opinions, annotators, and sources. We evaluate approaches to a popular argument summarization task called Key Point Analysis, which shows how these approaches struggle to (1) represent arguments shared by few people, (2) deal with data from various sources, and (3) align with subjectivity in human-provided annotations. We find that both general-purpose LLMs and dedicated KPA models exhibit this behavior, but have complementary strengths. Further, we observe that diversification of training data may ameliorate generalization in zero-shot cases. Addressing diversity in argument summarization requires a mix of strategies to deal with subjectivity.

## introduction getting an overview of the arguments concerning controversial issues is often difficult for those participating in ongoing discussions. this is because there are many points being communicated, no way to track which arguments were already encountered, and haphazard miscommunication or conflict. automatic summarization is a way to provide a comprehensible overview of the opinions @xcite . however, generating summaries representative of the arguments involved in a discussion is difficult @xcite . argument summarization extends beyond text summarization because it separates argumentative and non-argumentative content, preserves the argumentative structure, and provides explicit stances on a central claim or hypothesis. summarizing arguments is challenging in many contexts, but the potential impact is high. for instance, after summarizing the arguments from societal discussions, the extracted arguments may shape new policies and be used to justify decisionmaking @xcite . similarly, businesses depend on review data to find customer feedback, which can be used to steer product design @xcite . although arguments are often summarized by hand in practice (e.g., @xcite @xcite , recent developments in argument mining (am) allow automatic analysis of argumentative text @xcite . obtaining summaries that faithfully represent openended opinions requires careful evaluation, especially in sensitive contexts, e.g., summarizing citizen feedback @xcite . one approach for generating comprehensive summaries of arguments is key point analysis (kpa, @xcite . in kpa, a corpus of opinions is analyzed for the key points, those arguments that are salient and repeated multiple times. however, some aspects of the kpa experimental design misalign with respect to real-world applications. we illustrate these blind spots, in particular, when applied to summarizing online societal discussions. we highlight three dimensions of diversity that are central to empowering citizens' opinions at scale @xcite : (1) incorporating the long tail of opinions, (2) being robust in handling data from multiple sources, and (3) including diverse perspectives from annotators. how current kpa approaches deal with the above dimensions of diversity is unexplored. we conduct an empirical study of different argument summarization approaches by incorporating the standardized benchmark and two other datasets to experiment. we develop specific analyses to un-cover how kpa approaches fare on each dimension of diversity. in addition to the existing approaches, we use llms by prompting them to perform kpa, as they may be an attractive alternative to current models. applying kpa approaches across several datasets that vary in how they address diversity leads to mixed results. kpa performance degrades when dealing with low-frequency opinions, i.e., opinions repeated by relatively few individuals. further, we observe that kpa approaches disregard subjective interpretations among individual annotators. finally, they generalize poorly across data sources when used in transfer learning settings, though approaches reveal complementary merits across tasks. contributions (1) we critically examine three dimensions of diversity-of opinions, sources, and annotators-in the kpa setup. ( 2 ) we analyze the behavior of existing metrics on one existing and two newly adapted datasets. (3) we analyze multiple methods, including prompt-based llms, broadening the scope of methods that can perform kpa. ## related work we outline three lines of related work: key point analysis, opinion summarization, and diversity. ## method we formulate the kpa subtasks-key point generation (kpg) and key point matching (kpm). we then introduce the three dimensions of diversity and consider them when applying kpa. ## experimental setup we describe the data, kpa methods, and metrics involved in our experiments. we make our source code foot_0 and data (van der meer et al., 2024) publicly available. ## conclusion we perform a novel diversity exploration of different kpa approaches on three distinct datasets. by splitting kpa into two subtasks (kpg and kpm), we investigate each subtask, independently. first, we find that an llm-based approach works well for generating key points, but fails to match arguments to key points reliably. conversely, smaller fine-tuned models are better at matching arguments to key points but struggle to find good key points consistently. second, using a single training set yields poor generalization across datasets, showing that data source impacts a kpa approach's ability to generalize. diversification of training data leads to promising results. third, across all datasets, we see that existing methods for kpa are insensitive to long tail diversity, decreasing performance for key points supported by few arguments. finally, all models are insensitive to differences between individual annotators, disregarding subjective interpretations of arguments and key points. we showed how multiple aspects of diversity, a core principle when interpreting opinions, are not evaluated using the standard set of metrics. our analysis revealed interesting complementary strengths of the kpa approaches. future efforts could focus on addressing diversity, either by mining for minority opinions directly @xcite , or by identifying possibly subjective instances using socio-demographic information @xcite . further, models can be enhanced with subjective understanding @xcite , or work together with humans to jointly address some of the diversity issues @xcite .