From Key Points to Key Point Hierarchy: Structured and Expressive Opinion Summarization (2023)

Abstract: Key Point Analysis (KPA) has been recently proposed for deriving fine-grained insights from collections of textual comments. KPA extracts the main points in the data as a list of concise sentences or phrases, termed Key Points, and quantifies their prevalence. While key points are more expressive than word clouds and key phrases, making sense of a long, flat list of key points, which often express related ideas in varying levels of granularity, may still be challenging. To address this limitation of KPA, we introduce the task of organizing a given set of key points into a hierarchy, according to their specificity. Such hierarchies may be viewed as a novel type of Textual Entailment Graph. We develop ThinkP, a high quality benchmark dataset of key point hierarchies for business and product reviews, obtained by consolidating multiple annotations. We compare different methods for predicting pairwise relations between key points, and for inferring a hierarchy from these pairwise predictions. In particular, for the task of computing pairwise key point relations, we achieve significant gains over existing strong baselines by applying directional distributional similarity methods to a novel distributional representation of key points, and further boost performance via weak supervision.

## introduction many organizations face the challenge of extracting insights from large collections of textual comments, such as user reviews, survey responses, and feedback from customers or employees. current text analytics tools summarize such datasets via word clouds @xcite or key phrases @xcite , which are often too crude to capture fine-grained insights. multi-document summarization methods, on the other hand @xcite @xcite , do not quantify the prevalence of each point in the summary, and are not well-suited for representing conflicting views @xcite . ## thinkp: a dataset for key point hierarchies in this section we present thinkp, a benchmark dataset of key point hierarchies. to build thinkp, we first apply key point analysis to reviews of businesses and products from multiple domains ( §4.1). a kph is then constructed manually from the set of key points extracted for each business or product ( §4.2). we provide statistics on the resulting dataset, as well as qualitative analysis of the types of inferences it includes ( §4.3). ## automatic kph construction we use a two-step approach to automatically build a kph from a set of key points. in the first step, we predict directional scores between all pairs of key points ( §5.1). in the second step, we construct a hierarchy based on the local scores ( §5.2). ## evaluation predicting local pairwise relations. figure 2 compares the performance of the different local scoring methods ( §5.1). for each domain, we consider all the key point pairs in the dataset, and show the precision/recall curve and the area under the curve (auc) for each method. auc results are also summarized in to further assess the contribution of model combination in the weak labeling setting, we also tested a configuration in which the silver data is labeled by the nli model (denoted nli-wl). the results are shown on the last row of hierarchy construction. next, we compare different methods for constructing a kph from the set of local pairwise scores ( §5.2). we use the scores from the best performing local method, nli+bininc-wl, as found in the previous experiment. we use the f1 measure as defined in section 4.2.3 as our evaluation metric, similar to @xcite . since thinkp has no development set ( §4.3), we employ a leave-one-out scheme to tune the threshold τ . specifically, for each kpa summary s, we find the threshold that maximizes the f1 score of the three other kphs in the same domain and predict a kph for s using this threshold. we then compute the f1 score for the predicted kphs in each domain. we also evaluated the quality of the predicted relations using only the local scores, with a threshold determined via leave-one-out, as before (last row in ## conclusion we introduced key point hierarchies as a novel representation for structured, expressive opinion summaries. we explored several approaches for automatic hierarchy construction from a given set of key points, which were evaluated on a new benchmark dataset we developed for this task. we also proposed a novel distributional representation for key points, which we leveraged via weak supervision to achieve substantial improvement on the subtask of predicting pairwise key point relations. while our initial results are promising, there is still much room for improvement, and we hope that releasing our dataset would encourage the community to further promote this line of research.