Quantitative argument summarization and beyond: Cross-domain key point analysis (2020)

Abstract: When summarizing a collection of views, arguments or opinions on some topic, it is often desirable not only to extract the most salient points, but also to quantify their prevalence. Work on multi-document summarization has traditionally focused on creating textual summaries, which lack this quantitative aspect. Recent work has proposed to summarize arguments by mapping them to a small set of expert-generated key points, where the salience of each key point corresponds to the number of its matching arguments. The current work advances key point analysis in two important respects: first, we develop a method for automatic extraction of key points, which enables fully automatic analysis, and is shown to achieve performance comparable to a human expert. Second, we demonstrate that the applicability of key point analysis goes well beyond argumentation data. Using models trained on publicly available argumentation datasets, we achieve promising results in two additional domains: municipal surveys and user reviews. An additional contribution is an in-depth evaluation of argument-to-key point matching models, where we substantially outperform previous results.

## introduction the need for summarizing views, arguments and opinions on a given topic is common to many text analytics applications, across a variety of domains. some prominent examples for this type of data are responses to open-ended questions in surveys, user reviews on products and services, and posts in online discussion forums. we will hereafter refer to such utterances that express an opinion, view, argument, ask, or suggestion, collectively as comments. compressing such textual collections into short summaries relies on their inherent redundancy. the goal of multi-document summarization (mds) algorithms is to create short textual summaries from document clusters sharing the same topic. these summaries aim to capture most of the relevant information in the input clusters, while removing redundancies. however, in many cases we would also like to quantify the prevalence of each of the points included in the summary. for example, when analyzing the responses of a municipal survey, it would be desirable to let the policy makers know that the point "the city needs better public transportation" in the summary matches 8% of the comments, while the points "please consider increasing the number of parks, walking and biking trails." and "electric rates are too high" match 4% and 2% of the comments, respectively. the users may also want to drill down to view the comments that were mapped to a specific point in the summary. recently, @xcite proposed key point analysis as a summarization framework that meets the above desiderata, in the context of argument summarization. given a collection of arguments on some topic, their approach aims to match each argument to a short list of key points, defined as high-level arguments. in their work, key points were manually composed by an expert, while the matching of arguments to key points was done automatically. the current work promotes this line of research in two important respects. first, we develop a method for automatic key point extraction (section 3), allowing fully automatic key point analysis. our method first selects short, high quality comments as key point candidates. it then leverages previous work on argument-to-key-point matching to select a subset of the candidates that achieve high coverage of the data. we show that this relatively simple approach for key point extraction achieves results on argumentation data that are on par with human experts. the second major contribution of this work is demonstrating the applicability of key point analysis in additional domains beyond argumentation. we report promising results on two datasets: municipal surveys and user reviews. remarkably, the results are achieved using the same argument matching and argument quality models that were trained on argumentation data, and require only minimal parameter tuning, but no domain-specific labeled data. an additional contribution is an extensive comparison of pre-trained transformer models for argument matching, in terms of both accuracy and run time, which results in substantial improvement over the best results reported by bar-haim et al. (section 2). ## matching comments to key points the goal of key point analysis is to extract key points and to match comments to these key points. as mentioned in the previous section (and will be further detailed in the next section), our key point selection algorithm is also based on matching comments to key points, making it a critical component in our system. we build on the work of @xcite , who developed a large-scale labeled dataset for the task of matching arguments to key points. the dataset, termed argkp, contains about 24k (argument, key point) pairs, for 28 controversial topics. each of the pairs is labeled as matching/nonmatching. given a set of key points for a topic, an argument could be matched to one or more key points, or to none of them. the arguments in this dataset are a subset of a larger dataset, the ibm-argq-rank-30kargs dataset, which contains 71 topics, with stance and argument quality annotations for each argument @xcite . bar-haim et al. only experimented with bert @xcite as a supervised model for argument matching, which they trained on the argkp dataset. we aimed to improve their results by testing several more recent transformer-based pretrained models that were shown to substantially outperform bert on various tasks @xcite , and in particular on the related task of recognising textual entailment (rte). we used the hug-gingface transformers framework and fine-tuned four different models: bert-large-uncased @xcite (bert), xlnet-large-cased @xcite (xlnet), roberta-large @xcite (roberta) and albert-xxlarge-v1 @xcite ) (albert). 1 we ran 4-fold cross-validation on the argkp dataset, where each fold had a train set of 17 topics, development set (dev-set) of 4 topics and test set of 7 topics. the learning rate for each model was tuned based on the final training loss in one of the splits. this learning rate was then used in all four splits. the selected learning rates were 2e-5 for bert, 7e-6 for xlnet, 5e-6 for roberta and 1e-5 for albert. for choosing the number of epochs, we trained each model with 3 epochs and 9 epochs and selected the one that performed better on the dev-set. all models were better when trained for 9 epochs, except bert that was better when trained for 3 epochs. the evaluation results for these models with the above parameters are shown in for each fold, we selected the threshold t that maximizes the f1 score over the dev-set. the model that achieves the best f1 score is albert with an f1 score of 0.809. roberta is second best with an f1 score of 0.773. however, inference time of roberta is about 6 times faster than albert (run times for each model are detailed in appendix a). taking run time into account, we decided, for practical reasons, to use the roberta model in the rest of the experiments. we apply this model to arguments, as well as other types of comments in different domains. notably, both albert and roberta substantially outperform bert, which only reaches f1 score of 0.721 (similar to the f1 of 0.713, reported for bert by @xcite . ## key point extraction in addition to the matching of comments to given key points, we wish to extract the key points automatically from the set of comments, to enable fully-automatic key point analysis. extraction is performed in two steps: first, a set of key point candidates is selected from the comments and second, the most salient candidates are selected as key points. ## related work the task of multi-document summarization (mds) is defined as creating a summary that is both concise and comprehensive from multiple texts with a shared theme, e.g., news articles covering the same event @xcite . a major chal-lenge for applying supervised learning to mds has been the limited amount of available training data. most of the approaches applied to the task were extractive, operating over graph-based representations of sentences or passages @xcite . recently, @xcite proposed a method for creating a large-scale dataset from wikipedia (wikisum), which allowed training an abstractive neural model for this task. key point analysis adds a quantitative dimension that is not addressed by mds, by measuring the prevalence of each point in the summary. many of the works on opinion summarization take an alternative, sentiment-based approach. these works aim to identify the main aspects discussed in user reviews, and quantify the sentiment towards each of these aspects @xcite @xcite . however, as noted by @xcite , it is still hard for a user to understand why an aspect received a particular rating. as demonstrated in in computational argumentation, several works have focused on pairwise argument similarity and clustering @xcite @xcite . these works, however, did not attempt to create textual summaries from the resulting clusters. @xcite summarized argumentative discussions through the extraction of salient "points", where each point is a verb and its syntactic arguments. the current work also extracts points from argumentative data, but our goal is to go beyond textual summaries, by matching each key point to its corresponding sentences in the input data. similar to egan et al., we also experimented with extracting syntactic subtrees as key points, but found that this often results in incomplete sentences or omission of important information. selecting short, high quality sentences as key points was found to perform better in our experiments. the line of research that is most relevant to the current work deals with matching argumentative texts to predefined, short lists of manuallycomposed arguments or points @xcite @xcite . @xcite matched crowd-contributed arguments, taken from the dataset of @xcite , to key points composed by a debate expert. we used the labeled dataset developed by @xcite to train our comment matching model. as previously discussed, the main contributions we make to this line of work are (i) fully-automatic key point analysis, enabled by automatic key point extraction, and (ii) demonstrating the applicability of key point analysis to additional domains besides argumentation, including surveys and user reviews. furthermore, we were able to achieve promising results on these domains using models that were only trained on argumentation data. ## conclusion key point analysis is a novel framework for summarizing arguments, opinions and views. it provides both textual and quantitative view of the main points in the summarized data, and allows the user to interactively drill down from points to the actual sentences they cover. previous work only applied key point analysis in the context of argumentation data, and required a domain expert for writing the key points. the current work addresses both of the above limitations. first, we present an automatic method for key point extraction, which is shown to perform on par with a human expert. second, our work demonstrates the potential of key point analysis in multiple domains besides argumentation. furthermore, we show that the necessary knowledge for key point analysis, once acquired by supervised learning from argumentation data, can be successfully applied cross-domain, making it unnecessary to collect domain specific labeled data for each target domain. in future work, we would like to improve comment matching, e.g., by making it stance-aware. we also plan to experiment with sequence-tosequence neural models for generating key point candidates from comments.