Do You Hear The People Sing? Key Point Analysis via Iterative Clustering and Abstractive Summarisation (2023)

Abstract: Argument summarisation is a promising but currently under-explored field. Recent work has aimed to provide textual summaries in the form of concise and salient short texts, i.e., key points (KPs), in a task known as Key Point Analysis (KPA). One of the main challenges in KPA is finding high-quality key point candidates from dozens of arguments even in a small corpus. Furthermore, evaluating key points is crucial in ensuring that the automatically generated summaries are useful. Although automatic methods for evaluating summarisation have considerably advanced over the years, they mainly focus on sentence-level comparison, making it difficult to measure the quality of a summary (a set of KPs) as a whole. Aggravating this problem is the fact that human evaluation is costly and unreproducible. To address the above issues, we propose a two-step abstractive summarisation framework based on neural topic modelling with an iterative clustering procedure, to generate key points which are aligned with how humans identify key points. Our experiments show that our framework advances the state of the art in KPA, with performance improvement of up to 14 (absolute) percentage points, in terms of both ROUGE and our own proposed evaluation metrics. Furthermore, we evaluate the generated summaries using a novel set-based evaluation toolkit. Our quantitative analysis demonstrates the effectiveness of our proposed evaluation metrics in assessing the quality of generated KPs. Human evaluation further demonstrates the advantages of our approach and validates that our proposed evaluation metric is more consistent with human judgment than ROUGE scores.

## introduction automated summarisation of salient arguments from texts is a long-standing problem, which has attracted a lot of research interest in the last decade. early efforts proposed to tackle argument summarisation as a clustering task, implicitly expressing the main idea based on different notions of relatedness, such as argument facets @xcite , similarity @xcite and frames @xcite . however, they do not create easy-tounderstand summaries from clusters, which leads to unmitigated challenges in comprehensively navigating the overwhelming wealth of information available in online textual content. recent trends aim to alleviate this problem by summarising a large collection of arguments in the form of a set of concise sentences that describe the collection at a high-level-these sentences are called key points (kps). this approach was first proposed by @xcite , consisting of two subtasks, namely, key point generation (selecting key point arguments from the corpus) and key point matching (matching arguments to these key points). later work applied it across different domains @xcite , for example for product/business reviews @xcite . while this seminal work advanced the state of the art in argument summarisation, a bottleneck is the lack of large-scale datasets. a common limitation of such an extractive summarisation method, is that it is difficult to select candidates that concisely capture the main idea in the corpus from dozens of arguments. although @xcite suggested extracting key point candidates from the broader domain (e.g. selecting key point candidates from restaurant or hotel reviews when the topic is "whether the food served is tasty") to overcome this fundamental limitation, it is impractical to assume that such data will always be available for selection. an alternative, under-explored line of work casts the problem of finding suitable key points as abstractive summarisation. research work in this direction aims to generate key points for each given argument, without summarising multiple of them @xcite . as such, their approach rephrases existing arguments rather than summarising them. one possible reason for key point generation being under-explored, is the lack of reliable automated evaluation methods for generated summaries. established evaluation metrics such as rouge @xcite and bleu @xcite rely on the n-gram overlap between candidate and reference sentences, but are not concerned with the semantic similarity of predictions and gold-standard (reference) data. recent trends consider automated evaluation as different tasks, including unsupervised matching @xcite , supervised regression @xcite , ranking @xcite , and text generation @xcite . while these approaches model the semantic similarity between prediction and reference, they are limited to per-sentence evaluation. however, this is likely insufficient to evaluate the quality of multiple generated key point summaries as a whole. for instance, the two key points "government regulation of social media contradicts basic rights" and "it would be a coercion to freedom of opinion" essentially contain the same information as the reference "social media regulation harms freedom of speech and other democratic rights", but individually contain different pieces of information. in this work, we propose a novel framework for generative key point analysis, in order to reduce the reliance on large, high-quality annotated datasets. compared to currently established frameworks @xcite , we propose a novel two-step abstractive summarisation framework. our approach first clusters semantically similar arguments using a neural topic modelling approach with an iterative clustering procedure. it then leverages a pre-trained language model to generate a set of concise key points. our approach establishes new state-of-the-art results on an existing kpa benchmark without additional annotated data. results of our evaluation suggest that rouge scores that assess generated key points against gold standard ones do not necessarily correlate with how well the key points represent the whole corpus. the novel set-based evaluation metric that we propose, aims to address this. overall, the main contributions of this work are as follows: we propose a novel framework for key point analysis, depicted in figure 1 , which significolours illustrate the correspondences between arguments and key points. nodes in orange represent many-to-many matches, i.e., key points that are shared between both clusters. the input for key point generation (kpg) is composed of a single cluster from key point modelling (kpm) with its corresponding stance and topic. key point importance is measured by the size of the clusters. for example, key point: school uniforms are expensive ( yellow ) has an importance of 5 (including the argument that belongs to both clusters). ## related work argument summarisation: the field of argument summarisation has developed considerably in recent years. @xcite used an attention-based neural network to construct concise and fluent summaries of opinions in news editorials or social media. @xcite , focussing on web search, introduced an unsupervised extractive summarisation approach to generate argument snippets representing the key claim and reason. all of these efforts tackled single document summarisation where only one argumentative text is sum-marised at a time. the earliest multi-document summarisation work attempted to summarise argumentative discussions in online debates by extracting summaries in the form of salient "points", where a point is a verb and its syntactic arguments @xcite . however, their approach relies on lexical features that make it difficult to capture variability in claims that share the same meaning but are expressed differently. the work of @xcite and @xcite aimed to cluster semantically similar arguments. however, these efforts did not attempt to summarise these clusters, hence main points in the corpus remained implicit. recent work proposed key point analysis, which aims to extract salient points from a corpus of arguments, providing a textual and quantitative view of the data @xcite . @xcite contributed to the development of this framework by proposing a graph-based extractive summarisation approach. one common limitation of extractive summarisation methods, however, is that it is difficult to select key point candidates that truly capture salient points from dozens of arguments . @xcite used an abstractive summarisation method, where each single argument and its topic were used as input in order to generate summaries. a set of sentences which have the highest scores based on rouge @xcite ranking, is then selected as key points. however, in practice this is not feasible as the computation of rouge scores requires the availability of gold standard key points. automatic evaluation of generated summaries: most of the current work relies on humancentric evaluation methods @xcite @xcite . however, they are time-consuming, costly and difficult to replicate. some of the work attempts to use automated evaluation methods such as rouge, a metric widely used to evaluate automatically generated summaries @xcite . this type of automatic metric compares generated sentences with gold standard ones, but it is difficult to measure their accuracy and effectiveness in terms of capturing semantic similarity. recent trends consider automated evaluation as different tasks. @xcite proposed unsupervised matching metrics, aimed at measuring semantic equivalence by mapping candidates and references to a distributed representation space. @xcite presented a supervised learning evaluation metric that can model human judgments by a novel pre-training scheme. their work demonstrates that pre-training a metric on task-specific synthetic data, before finetuning it on handpicked human ratings can improve metric robustness. @xcite considered the problem as a ranking task, leveraging breakthroughs in multilingual pre-trained models to generate ratings that resemble human judgments. @xcite instead suggested that evaluating the quality of summaries can be treated as a text generation task. the main idea is that converting a well-performing generated text to/from a reference text would easily achieve higher scores. while these approaches have advanced the field, they all focus on sentence-level evaluation. our task, however, requires the evaluation of a set of key points. the reason is that when comparing generated key points to gold-standard annotations at a sentence level, important information could be lost. this can only be retained by considering all sentences at once. ## methodology in this section, we describe our framework in detail. as can be seen from figure 1 , for each debate topic such as "should we abandon the use of school uniforms?", we take a corpus of relevant arguments grouped by their stance towards the topic (i.e. "pro" or "con") as input, as mined from online discussion boards. as part of kpm, these arguments are clustered using a neural topic modelling approach to group them by their common theme. the clusters are then used as input to the kpg model for summarisation, which is optimised to generate a key point for each argument cluster. during the training of our model for kpm, we employ data augmentation. ## experimental setup broadly speaking, we aim to investigate the efficacy of our proposed kpm framework as well as the evaluation metrics. specifically, we ask: (i) does the proposed approach improve the performance of the task? (ii) does data augmentation help with the lack of training data? (iii) does the re-clustering of outliers by using ic improve performance on downstream tasks? (iv) does the proposed evaluation framework correlate better with human judgments than raw rouge scores? to answer question (i) we compare the performance of our proposed approach to established previous approaches on the task of kpa. for questions (ii) and (iii), we perform ablation studies to measure the impact our using supervised and unsupervised kpm pipelines (s-kpm and us-kpm) as well as data augmentation (+da) and iterative clustering (+ic). for question (iv), we conduct manual evaluation. baselines: we compare our approach with previous known and open-source work-enigma @xcite and graph-based summarization (gbs) @xcite foot_2 , selecting their best reported results as the baseline. enigma uses an abstract summarisation approach, employing pegasus @xcite as the summarisation model, to generate candidate kps by taking a single argument and its corresponding topic as input. finally, the top-n highest rouge scores with reference kps were selected as the final result. similar to the work of @xcite , gbs constructs an undirected graph with arguments as nodes. nodes with sufficiently high argument quality scores @xcite , and node matching scores @xcite are connected. the importance score of each argument is then calculated based on pagerank @xcite and ranked in descending order. finally, only those arguments where matching scores are below the threshold of the already selected candidates are added to the final set of key points. evaluation metrics: we calculate rouge @xcite scores on the test set, by comparing the concatenation of all generated key points to the concatenation of the reference, averaging for all topic and stance combinations. furthermore, in order to evaluate the quality of the generated key points invariant to the order of sentences, we also compare the performance based on the proposed set-level evaluation approach. similar to our idea, the earth mover's distance (emd) @xcite is a measure of the similarity between two data distributions. by combining word mover's distance (wms) @xcite and sentence mover's similarity (sms) @xcite , sentence + word mover's similarity (s+wms) measures both the word distribution of a single sentence and similarity at the set level. however, an observable shortcoming is that they consider a set of sentences as a single paragraph, without splitting and using glove embeddings @xcite instead of fine-tuning on sentence-level similarity. human evaluation: taking into account the wealth of problems arising from automatically evaluating generated texts, we further verify the reliability of our obtained results,by means of human evaluation. seven annotators were selected, all of whom are graduate students with a diploma from a university in the uk. before starting, all annotators received task-oriented training, the specific instructions can be found in appendix c.1. after an introduction, they had to answer a questionnaire containing 66 questions for all topics and stances in the test set. the annotators were asked to answer on a likert scale ranging from "very good" (5) to "not good at all" (1). the first evaluation task (ht1) investigates how well the generated summaries of clusters serve as kps. following bar-haim et al. (2021), we assessed the quality of the key points in four requirements: validity, sentiment, informative-ness and singleaspect. annotators are asked to read three sets of kps separately (reference, our best approach, previous work), assigning each of the four dimensions above a single score, and then ranking each of the three outputs the outputs from best to worst. @xcite , as their code is not open source. 11b, 3b, large and base refer to flan-t5-xxl, flan-t5-xl, flan-t5-large and t5-effictive-base, respectively. s+wms stands for sentence + word mover's similarity @xcite . ## results and analysis proposed approach improves performance on kpa task: our proposed two-step method outperforms the reference implementations on the full kpa task, with improvements of up to 12% and 14% in rouge and soft-f1, respectively, as shown in overall, each proposed improvement (+da and +ic) contributes to achieve better scores. a robustness experiment was then performed on the best-performing approach, with 10 runs, showing that the overall performance is still up to 11% superior compared to the baseline according to rouge, and up to 3% superior based on the proposed evaluation approach. it is worth noting that unsupervised kpm with ic (us-kpm+ic) yields increases of more then ten points in rouge-l and two soft-f1 (bleurt) percent points compared to the best performing baseline, demonstrating that the proposed method outperforms previous state-of-the-art approaches even without training the clustering model and relying on data augmentation. our human evaluation further supports these findings: in the ranking task t1, our method was ranked higher than the baselines, slightly behind human-written reference kps. as can be seen from ## conclusion this paper contributes to the development of key point analysis. firstly, we proposed a two-step abstractive summarisation framework. compared with previous work, our approach achieves performance on par with a human without additional training samples. secondly, we developed a new evaluation toolkit, whose effectiveness was demonstrated with human annotators, presenting a more realistic view of the generated kps' quality than traditional automatic evaluation metrics. in future work, we will address the issue that kps with few matching arguments are difficult to cluster, by using contrastive learning @xcite to facilitate better intra-cluster and inter-cluster distances.