Enhancing Argument Summarization: Prioritizing Exhaustiveness in Key Point Generation and Introducing an Automatic Coverage Evaluation Metric (2024)

Abstract: The proliferation of social media platforms has given rise to the amount of online debates and arguments. Consequently, the need for automatic summarization methods for such debates is imperative, however this area of summarization is rather understudied. The Key Point Analysis (KPA) task formulates argument summarization as representing the summary of a large collection of arguments in the form of concise sentences in bullet-style format, called key points. A sub-task of KPA, called Key Point Generation (KPG), focuses on generating these key points given the arguments. This paper introduces a novel extractive approach for key point generation, that outperforms previous state-of-the-art methods for the task. Our method utilizes an extractive clustering based approach that offers concise, high quality generated key points with higher coverage of reference summaries, and less redundant outputs. In addition, we show that the existing evaluation metrics for summarization such as ROUGE are incapable of differentiating between generated key points of different qualities. To this end, we propose a new evaluation metric for assessing the generated key points by their coverage. Our code can be accessed online.

## introduction controversial issues that divide people are prevalent on social media platforms. these platforms provide fertile ground for individuals to express their arguments and engage in debates, leading to a vast amount of textual opinions expressed in diverse manners. whether it's policymakers, businesses, or individuals, every decision-maker can benefit from a synthesized version that summarizes the main distinct key points related to an issue. a key point (kp) should be a concise, high-level argument that aligns with a significant facet of a recurrent argument, while still being informative. the work of @xcite and @xcite were the first to formulate the key point analysis (kpa) task and to introduce a corresponding dataset, the argkp dataset. the kpa shared task was comprised of two tracks, key point matching (kpm) and key point generation (kpg). the goal of kpm is to map the correct key point to arguments, i.e. predicting whether argument-key point pairs match or not. in kpg, the goal is to generate the key points, given the arguments as inputs. in the kpa task, the term "generated key points" refers to output summary generated by the model. the model itself, whether abstractive or extractive, is referred as a kpg model @xcite . the top two previous approaches to kpg, as evaluated by the task organizers, were @xcite and @xcite . however, due to their construction that prioritizes the most popular recurrent arguments, both approaches often generate text that emphasizes redundant key points. additionally, both methods rely on filtering low-quality arguments, which limit the input. this compromises the potential coverage of most or all key points, as less popular ones have a limited distribution, and further filtering may further diminish their chances of representation. moreover, @xcite select only short arguments for inclusion in a summary. this of-ten yields only a few arguments from an input of around two hundred and produces no summary for inputs of a hundred or fewer. in this work, we introduce a novel and simple framework for key point generation. it is versatile, catering to both small and large numbers of input arguments. most critically, it generates a summary comprised of self-contained arguments that cover the majority of existing key points (in a reference summary) with minimal redundancy. in this work we use key point generation and argument summarization interchangeably as the goal of the kpg model is to summarize a group of arguments. on another aspect, the kpa task organizers relied on human judges to score and rank the kpg models. judges were instructed to score output summaries based on their coverage of reference summaries and redundancy. although human evaluation is accurate, it is neither scalable nor reproducible. alternatively, automatic evaluation metrics, such as rouge @xcite , are often used for evaluating generated summaries. however, as shown by our experiments, they are not as effective for evaluating kpg task. hence, in addition to the argument summarization approach, we propose a new evaluation metric for coverage based on the kpa shared task. for the summarization of arguments, we employ an extractive clustering based approach. using semantic-based clustering, we group similar arguments together and use a matching model to find the argument that best represents the cluster. the representatives of clusters are chosen as the summary. this approach allows less popular key points, i.e. smaller clusters, to be represented. moreover, the matching model used in argument selection eliminates the need for a filtering step that not only limits the arguments that can be extracted, but also introduces additional error and significantly slows down the process. we also demonstrate the versatility of our model by comparing it to previous approaches on datasets other than the traditional argkp, which the earlier methods primarily utilized. to automatically assess the coverage of argument summaries, we utilize a key point matching (kpm) model to compute a coverage score. this model determines whether a given produced argument matches with a reference or ground truth key point. the coverage score quantifies the proportion of matched reference key points in the summary. furthermore, by enforcing a limit on the number of generated key points, a summary with high coverage of distinct key points would indicate low redundancy. our experiments confirm the efficacy and appropriateness of the proposed metric compared to traditionally employed ones. in short, our contributions are as follows: ## conclusion this paper proposes a new extractive method for kpg that aims to generate key points with a high coverage of reference key points and less redundancy. our method adopts a clustering-based approach with two distinct selection methods to achieve this goal. our experiments show that the proposed methods outperform previous sota in terms of coverage, redundancy, and quality . lastly, we proposed a metric for coverage, that outperforms previous metrics on coverage prediction.