Identifying Helpful Sentences in Product Reviews (2021)

Abstract: In recent years online shopping has gained momentum and became an important venue for customers wishing to save time and simplify their shopping process. A key advantage of shopping online is the ability to read what other customers are saying about products of interest. In this work, we aim to maintain this advantage in situations where extreme brevity is needed, for example, when shopping by voice. We suggest a novel task of extracting a single representative helpful sentence from a set of reviews for a given product. The selected sentence should meet two conditions: first, it should be helpful for a purchase decision and second, the opinion it expresses should be supported by multiple reviewers. This task is closely related to the task of Multi Document Summarization in the product reviews domain but differs in its objective and its level of conciseness. We collect a dataset in English of sentence helpfulness scores via crowd-sourcing and demonstrate its reliability despite the inherent subjectivity involved. Next, we describe a complete model that extracts representative helpful sentences with positive and negative sentiment towards the product and demonstrate that it outperforms several baselines.

## introduction customer reviews are known to be a valuable source of information for potential buyers. this is evident from the high engagement of customers with reviews, for example by up-voting a review for its helpfulness. 1 as online shopping platforms attract more traffic it is becoming increasingly difficult to consume the wealth of information customers share. for this reason, helpful reviews (defined as such by costumers) are made more visible than those that are less helpful. the topic of review helpfulness has attracted a lot of academic interest in which reviews were always considered as a whole (see @xcite for a survey). however, in some scenarios, such as the limited real-estate in mobile screens, or in voice interactions with a virtual assistant, presenting a full review is impractical and the need to automatically extract helpful excerpts arises. while in the mobile scenario, a persistent customer may still be able to read the entire review, the voice scenario is inherently more challenging as it demands patience and focus from the customer, while the assistant reads the text out loud. as a result, the need for extreme brevity and the ability to understand what matters most to customers becomes crucial. in addition to brevity and helpfulness, another desirable property from the extracted content is being faithful to the reviews as a whole. indeed, a costumer looking for relevant and helpful reviews, often interacts with more than one review before making their decision, trying to pinpoint those helpful bits of information that are shared by multiple reviewers. this process is tedious because of the sheer amount of reviews and biased because of the order they appear in. a system that aims to replace this process while maintaining trust in the content it provides should be able to extract concise helpful texts that repeat across multiple reviews, indicating that they are faithful to the reviews' content (from here onward we shall refer to such texts as "faithful"). our goal is to extract such sentences, i.e., sentences that are both helpful for a purchase decision and faithful. to this end, we first define two new notions: a helpful sentence is a sentence which is considered helpful by the average customer in their purchase decision process. a representative helpful sentence (rhs) is a helpful sentence that is also highly supported, that is, the ideas it expresses appear in multiple reviews for the given product (not necessarily in the exact same wording). it is traditionally assumed that judging the importance of a text excerpt requires reading the entire text. we challenge this assumption, at least in the domain of product reviews, and collect a dataset of single review sentences with their helpfulness scores by averaging the scores assigned to them by multiple crowd workers. we show that despite the highly subjective nature of this task, and despite the fact that workers are exposed to sentences without their surrounding context, the resulting scores are reliable. using the data we collected, from 6 different categories, ranging from electronics to books, we train and evaluate several supervised algorithms to predict helpfulness score, which achieve promising results. finally, we present an initial implementation of a model that given a set of product reviews, extracts a single positive rhs (supports the purchase) and a single negative rhs (opposes the purchase). in summary, the main contributions of this work are: (1) we propose a novel task that given a set of reviews for a product, outputs a single sentence that is both helpful for a purchase decision and supported by multiple reviewers; (2) we show that the helpfulness of a sentence can be reliably rated based solely on the sentence, allowing for an efficient dataset creation. these helpfulness scores can be leveraged for other tasks such as highlighting important parts of a review; (3) we publish a novel dataset of sentences taken from customer reviews along with their helpfulness score; foot_1 (4) we develop an end-to-end model for our task that shows promising results and outperforms several baselines. ## related work review helpfulness modeling and prediction customer reviews are a valuable source of information for customers researching a product before making a purchase @xcite . diaz and ng (2018) survey recent work on the tasks of modeling and predicting review helpfulness. while some researchers treat helpfulness votes as groundtruth, others have argued that these votes are not good indicators for actual review helpfulness @xcite @xcite . some general observations have been made based on helpfulness votes, e.g., review length has been shown to be strongly correlated to helpfulness @xcite @xcite @xcite . another widely-agreed indication for review helpfulness is the review star rating @xcite @xcite . a related dataset was presented in @xcite . the main advantages of the dataset we create over this previously suggested one are: (1) binary vs. continuous scores -we use continuous scores rather than binary scores. our aim is to surface the most helpful sentences, which is not possible if many of the sentences are annotated as equally helpful; (2) range of products/domains -the previous dataset includes only 5 products, all from the electronics domain. our dataset is significantly more diverse, providing annotations for 123 products from 6 different domains, allowing to evaluate a model's ability to generalize across domains. product review summarization the most common approach for product review summarization, which centers the summary around a set of extracted aspects and their respective sentiment, is termed aspect based summarization. one of the early abstractive works, by @xcite , was designed to output lists of aspects and sentiments. other works target a traditional summarization output and at times somewhat simplify the task by assuming aspects or seed words are provided as input @xcite @xcite . recently advances were made on unsupervised abstractive reviews summarization, by leveraging neural networks @xcite followed by a few shot variant @xcite . extractive summarization include earlier works such as @xcite ; @xcite and @xcite who suggested to use review helpfulness votes as means to improve the content extraction process. more recently, @xcite suggested a novel generative topic aspect sentiment model. ## representative helpful sentences task definition in this work, we focus on summarization of reviews in the setting of shopping over voice with the help of a virtual assistant. our goal is to provide users with content that is both helpful and faithful in this challenging setting where the information the user can absorb is extremely limited. first, we aim to maximize the informativeness, while maintaining brevity. to this end, we introduce a new notion of helpful sentences -sentences which the average costumer will consider as helpful for making a purchase decision. next, to ensure faithfulness, we introduce the notion of support for a given sentence -the number of review sentences with a highly similar content. we seek to automatically identify a helpful sentence with a wide support, which we term representative helpful sentence (rhs). note that representative helpful sentences, being supported by many similar sentences, are by construction faithful to the review pool from which they are extracted. we restrict ourselves to single sentences that are extracted as-is from product reviews, as this serves as another mechanism to ensure faithfulness. we do not restrict the number of reviews in the input. ## helpful sentences annotation our main contribution in this work lies in the notion of helpful sentences and the ability to identify such sentences without observing entire reviews. in what follows, we describe the process of compiling a dataset of sentences along with their helpfulness scores using crowdsourcing. note that this dataset is intended solely for scoring helpfulness of sentences. faithfulness is ensured by other means which are not reflected in the dataset, i.e. by requiring a rhs to have a wide support of similar sentences, as discussed in section 3 and implemented in our model, as described in section 5. ## surfacing representative helpful sentences we now turn to create an end-to-end model for surfacing representative helpful sentences (rhs): given a set of reviews for a certain product, we aim to output a single rhs with positive sentiment and a single rhs with negative sentiment. figure 1 depicts the different sub-components of our model. given a set of reviews, we preprocess the input and predict helpfulness scores for each of the sentences. next, we analyze the sentiment of each sentence and separate into positive and negative sets. following that, the support of each sentence is determined, and finally we select the rhs sentence based on its helpfulness score and its support. in what follows, we describe each of the components in details. preprocessing we remove html tags and split the cleaned reviews into sentences. the sentences are then filtered by removing sentences of extreme length (both short and long). see section d in the appendix for additional details. helpfulness estimation this component assigns a helpfulness score for each sentence and removes all sentences with score below 1. this filtering serves two purposes: first, it ensures that we do not output any sentence in case there is no helpful sentence in the product reviews. second, it reduces the runtime of the downstream similarity and support component which is quadratic in the number of sentences. we experiment with three helpfulness models and find that a pre-trained bert @xcite fine-tuned on our training data performs best. the two other models we compare are: (1) tf-idf: a model that treats each sentence as a bagof-words. we use tfidfvectorizer from the sklearn package to convert each sentence into a vector and then fit a ridge regression model on top of it; (2) st-ridge: a model that fits a ridge regression on top of the sentence-transformers embedding @xcite . we use 3 measures for evaluation: mean squared error (mse), which is the traditional measure for regression, pearson correlation between the predicted score and the ground-truth score, and finally a ranking measure that evaluates the quality of the top ranked sentence (ndcg@1). the results are depicted in ## evaluation the evaluation of our end-to-end model is challenging and does not have a natural scheme. recall that we do not restrict our input to small random samples of the review set, as commonly done in review summarization, and was shown to produce biased results @xcite . instead, we allow for dozens or hundreds of reviews per product. thus, we cannot expect annotators to carefully read the full input before choosing an rhs. nonetheless, we show that our notion of helpfulness is indeed useful for surfacing important review content by comparing our models to previous summarization works in two different settings. ## conclusion in this paper we address the challenge of summarizing product reviews with limited space, like when using a virtual assistant. we define a new notion that fits the needs of this setting, a representative helpful sentence, and propose a new task accordingly: given a set of product reviews, extract a sentence that is both helpful for a purchase decision and well supported by the opinions expressed in the reviews. as a first step, we collect and annotate a new dataset of review sentences with their helpfulness scores, and make this dataset available to facilitate further research. next, we develop an endto-end model for surfacing representative helpful sentences. our model combines several necessary components which are optimized for our goal. in order to get a feeling for the performance of our model, we compare our results to summarization tasks that are similar in nature, and show that our model performs better in the aspects we target.