Every Bite Is an Experience: K ey P oint A nalysis of Business Reviews (2021)

Abstract: Previous work on review summarization focused on measuring the sentiment toward the main aspects of the reviewed product or business, or on creating a textual summary. These approaches provide only a partial view of the data: aspect-based sentiment summaries lack sufficient explanation or justification for the aspect rating, while textual summaries do not quantify the significance of each element, and are not well-suited for representing conflicting views. Recently, Key Point Analysis (KPA) has been proposed as a summarization framework that provides both textual and quantitative summary of the main points in the data. We adapt KPA to review data by introducing Collective Key Point Mining for better key point extraction; integrating sentiment analysis into KPA; identifying good key point candidates for review summaries; and leveraging the massive amount of available reviews and their metadata. We show empirically that these novel extensions of KPA substantially improve its performance. We demonstrate that promising results can be achieved without any domain-specific annotation, while human supervision can lead to further improvement.

## introduction with their ever growing prevalence, online opinions and reviews have become essential for our everyday decision making. we turn to the wisdom of the crowd before buying a new laptop, choosing a restaurant or planning our next vacation. however, this abundance is often overwhelming: reading hundreds or thousands of reviews on a certain business or product is impractical, and users typically have to rely on aggregated numeric ratings, complemented by reading a small sample of reviews, which may not be representative. the vast majority of available information is left unexploited. opinion summarization is a long-standing challenge, which has attracted a lot of research interest over the past two decades. early works @xcite @xcite @xcite aimed to extract, aggregate and quantify the sentiment toward the main aspects or features of the reviewed entity (e.g., food, price, service, and ambience for restaurants). such aspectbased sentiment summaries provide a high-level, quantitative view of the summarized opinions, but lack explanations and justifications for the assigned scores @xcite ). an alternative line of work casts this problem as multi-document summarization, aiming to create a textual summary from the input reviews @xcite @xcite . while such summaries provide more detail, they lack a quantitative view of the data. the salience of each element in the summary is not indicated, making it difficult to evaluate their relative significance. this is particularly important for the common case of conflicting opinions. in order to fully capture the controversy, the summary should ideally indicate the proportion of favorable vs. unfavorable reviews for the controversial aspect. recently, key point analysis (kpa) has been proposed as a novel extractive summarization framework that addresses the limitations of the above approaches @xcite . kpa extracts the main points discussed in a collection of texts, and matches the input sentences to these key points @xcite . the salience of each kp corresponds to the number of its matching sentences. the set of key points is selected out of a set of candidatesshort input sentences with high argumentative quality, so that together they achieve high coverage, while aiming to avoid redundancy. the resulting summary provides both textual and quantitative positive key points % reviews negative key points % reviews amazingly helpful and friendly staff. ## data and task in this work we apply kpa to business reviews from the yelp open dataset foot_2 . the dataset contains about 8 million reviews for 200k businesses. each business is classified into multiple categories. restaurants is by far the most common category, comprising the majority of the reviews. besides restaurants, the dataset contains a wide variety of other business types, from nail salons to dentists. we focus on two business categories in our experiments: restaurants (4.9m reviews) and hotels (258k reviews). we will henceforth refer to these business categories as domains. each review includes, in addition to the review text, several other attributes, most relevant for our work is the "star rating" on a 1-5 scale. we filtered and split the dataset as follows. first, we removed reviews with more than 15 sentences (10% of the reviews). second, we removed businesses with less than 50 reviews. the remaining businesses were split into train, development (dev) and test set, as detailed in our goal is to create a summary of the reviews for a given business. the summary would list the top k positive and top k negative kps, and indicate for each kp its salience in the reviews, represented by the percentage of reviews that match the kp. a review is matched to a kp if at least one of its sentences is matched to that kp. an example of such summary is given in ## classification models our system employs several classification models: in addition to the matching and argument quality models discussed in section 2, in this work we add a sentiment classification model and a kp quality model, to be discussed in the next sections. all four classifiers were trained by fine-tuning a roberta-large model @xcite . prior to the fine-tuning of each classifier, we adapted the model to the business reviews domain, by pretraining on the yelp dataset. we performed masked lm pertraining @xcite on 1.5 million sentences sampled from the train set with a length filter of 20-150 characters per sentence. the following parameters were used: learning rate -1e-5; 2 epochs. training took two days on a single v100 gpu. the matching model was then obtained by fine-tuning the pre-trained model on the argkp dataset, with the parameters specified by @xcite . the quality model was fine-tuned following the procedure described by @xcite , except for using roberta-large instead of bertbase, with learning rate of 1e-5. ## incorporating sentiment into kpa previous work on kpa has ignored the issue of sentiment (or stance) altogether. when applied to argumentation data, it was assumed that the stance of the arguments is known, and kpa was performed separately for pro and con arguments. accordingly, the argkp dataset only contains (argument, kp) pairs having the same stance. there are, however, several advantages for incorporating sentiment into kpa, in particular when analyzing reviews: we developed a sentence-level sentiment classifier for yelp data by leveraging the abundance of available star ratings for short reviews. we extracted from the entire train set reviews having at most 3 sentences and 64 tokens. reviews with 1-2, 3 and 4-5 star rating were labeled as negative (neg, 20% of the reviews), neutral (neut, 11%) and positive (pos, 69%), respectively. the reviews were divided into a training set, comprising 235,481 reviews, and a held-out set, comprising 26,166 reviews. the sentiment classifier was trained by finetuning the pre-trained model on the above training data, for 3 epochs. the first two rows in since we ultimately wish to apply the classifier to individual sentences, we also annotated a small sentence-level benchmark of 158 reviews from the held-out set, which contain 952 sentences. we selected a minimal threshold t s for predicting pos or neg sentiment. if both pos and neg predictions are below this threshold, the sentence is predicted as neut. the threshold was selected so that the recall of both pos and neg is at least 70%, while aiming to maximize precision foot_3 . sentence-level performance on the benchmark using this threshold is shown in the last two rows of we integrate sentiment into kpa as follows. we extract positive kps from a set of sentences classified as positive, and likewise for negative kps. in order to further improve precision, positive (negative) sentences are only selected from positive (negative) reviews. when matching sentences to the extracted kps we filter out neutral sentences and match sentences only to kps with the same polarity. however, at this stage we do not filter by the review polarity, since we would like to allow matching positive sentences in negative reviews and vice versa, as well as positive and negative sentences in neutral reviews. ## collective key point mining kpa is an extractive summarization method: kps are selected from the review sentences being summarized. when generating a summary for a business with just a few dozens of reviews, the input reviews may not have enough good kp candidatesshort sentences that concisely capture salient points in the reviews. this is a common problem for extractive summarization methods, where it is often difficult to find sentences that fit into the summary in their entirety. we propose to address this problem by mining kps collectively for the whole domain (e.g., restaurants or hotels). the extracted set of domain kps is then matched to the review sentences of each analyzed business. this method can extract kps from reviews of thousands of businesses, rather than from a single business, and therefore is much more robust. it overcomes a fundamental limitation of extractive summarization -limited selection of candidate sentences, while sidestepping the com- for each domain, we sampled 12,000 positive reviews and 12,000 negative reviews from the train set, from which positive and negative kps were extracted, respectively foot_4 . we extracted positive and negative sentences from the reviews using the sentiment classifier, as described in the previous section. we filtered sentences with less than 3 tokens or more than 36 tokens (not including punctuation), as well as sentences with less than 10 characters. the number of positive and negative sentences obtained for each domain is detailed in the number of roberta predictions required by the algorithm is o(#kp-candidates × #sentences). while the input size in previous work was up to a few thousands of sentences, here we deal with 50k-60k sentences per run. in order to maintain reasonable run time, we had to constrain both the number of sentences and the number of kp candidates. we selected the top 25% sentences with the highest quality score. the maximal number of kp candidates was 1.5 × √ n s , where n s is the number of input sentences, and the highest-quality candidates were selected. each run took 3.5-4.5 hours using 10 v100 gpus. ## improving key point quality previous work did not attempt to explicitly define the desired properties kps should have, or to de-velop a model that identifies good kp candidates. instead, kp candidates were selected based on their length and argument quality, using the quality model of @xcite . this quality model, however, is not ideally suited for selecting kp candidates for review summarization: first, it is trained on crowd-contributed arguments, rather than on sentences extracted from user reviews. second, quality is determined based on whether the argument should be selected for a speech supporting or contesting a controversial topic, which is quite different from our use case. we fill this gap by defining the following requirements from a kp in review summarization: the kp should also be general enough to be relevant for other businesses in the domain. a common example of sentences that are too specific is mentioning the business name or a person's name ("byron at the front desk is the best!"). as we show in section 8, the method presented in the previous sections extracts many kps that do not meet the above criteria. in order to improve this situation, we developed a new kp quality classifier. we created a labeled dataset for this task, as follows. we sampled from the restaurant and hotel reviews in the train set 2,000 sentences comprising 3-8 tokens and minimal argument quality of t quality . each sentence was annotated for each of the above criteria 7 by 10 crowd annotators, using the appen platform 8 . we took several measures 7 the guidelines are included in the appendix. 8 https://appen.com/ to ensure annotation quality, following @xcite and @xcite . first, the annotation was performed by trusted annotators, who performed well on previous tasks. second, we employed the annotator-κ score @xcite , which measures inter annotator agreement, and removed annotators whose annotator-κ was too low. the details are provided in the appendix. for each sentence and each criterion, the fraction of positive annotations was taken to be its confidence. the final dataset was created by setting upper and lower thresholds on the confidence value of each of the four criteria. sentences that matched all the upper thresholds were considered positive. sentences that matched any of the lower thresholds were considered negative. the rest of the sentences were discarded. the threshold values we used are given in the appendix. overall, the dataset contains 404 positive examples and 1,291 negative examples. we trained a kp quality classifier by fine-tuning the pretrained roberta model (cf. section 4) on the above dataset (4 epochs, learning rate: 1e-05). figure 1 shows that this classifier (denoted kp quality ft) performs reasonably well on the dataset, in a 4-fold cross-validation experiment. unsurprisingly, the argument quality classifier trained on argumentation data is shown to perform poorly on this task. the classifier was used to filter bad kp candidates, as part of the kp mining algorithm (section 6). candidates that passed this filtering were filtered and ranked by the argument quality model as before. we selected a threshold of 0.4 for the classifier, which corresponds to keeping 32% of the candidates, with precision of 0.62 and recall of 0.82. ingly defined coverage as the fraction of matched sentences. we are more interested in review-level coverage, since not all the sentences in the review are necessarily relevant for the summary. given kpa results for b, k and t match , we can compute the following measures: computing precision requires a labeled sample. we create a sample s by repeating the following procedure until n samples are collected: ## related work previous work on review summarization was dominated by two paradigms: aspect-based sentiment summarization and multi-document opinion summarization. aspect-based sentiment summarization. this line of work aims to create structured summaries that assign an aggregated sentiment score or rating to the main aspects of the reviewed entity @xcite @xcite @xcite . aspects typically comprise 1-2 words (e.g., service, picture quality), and are either predefined or extracted automatically. a core sub-task in this approach is aspect-based sentiment analysis: identification of aspect mentions in the text, which may be further classified into highlevel aspect categories, and classification of the sentiment towards these mentions. recent examples are @xcite @xcite . the main shortcoming of such summaries is the lack of detail, which makes it difficult for a user to understand why an aspect received a particular rating @xcite . although some of these summaries include for each aspect a few supporting text snippets as "evidence", these examples may be considered anecdotal rather than representative. ## conclusion we introduced a novel paradigm for summarizing reviews, based on kpa. kpa addresses the limitations of previous approaches by generating summaries that combine both textual and quantitative views of the data. we presented several extensions to kpa, which make it more suitable for large-scale review summarization: collective key point mining for better key point extraction; integrating sentiment analysis into kpa; identifying good key point candidates for review summaries; and leveraging the massive amount of available reviews and their metadata. we achieved promising results over the yelp dataset without requiring any domain-specific annotations. we also showed that performance can be substantially improved with human supervision. while we focused on user reviews, the methods introduced in this work may improve kpa performance in other domains as well. in future work we would like to generate richer summaries by combining domain level key points with "local" key points, individually extracted per business. it would also be interesting to adapt current methods for unsupervised abstractive summarization to generate key points.