Ranking Online Reviews Based on Their Helpfulness: An Unsupervised Approach (2021)

Abstract: Online reviews are an essential aspect of online shopping for both customers and retailers. However, many reviews found on the Internet lack in quality, informativeness or helpfulness. In many cases, they lead the customers towards positive or negative opinions without providing any concrete details (e.g., very poor product, I would not recommend it). In this work, we propose a novel unsupervised method for quantifying helpfulness leveraging the availability of a corpus of reviews. In particular, our method exploits three characteristics of the reviews, viz., relevance, emotional intensity and specificity, towards quantifying helpfulness. We perform three rankings (one for each feature above), which are then combined to obtain a final helpfulness ranking. For the purpose of empirically evaluating our method, we use review of four product categories from Amazon review. The experimental evaluation demonstrates the effectiveness of our method in comparison to a recent and state-of-the-art baseline.

## introduction reviews are an essential aspect of information that allows users to obtain insight into a product of interest before purchasing. typically, users write their reviews in order to express their satisfaction or dissatisfaction about purchased items or services. products and sellers with more positive reviews tend to gain more new customers than products or sellers without reviews or with many negative reviews. this is because customers feel more confident buying products that have been recommended by other buyers. popular products could have hundreds or thousands of reviews, which makes it impossible for the customers to read all of them. moreover, it is not easy for a user to prioritize reading the most informative reviews since there are 1 http://jmcauley.ucsd.edu/data/amazon/ often no such ranking options. some websites rank reviews based on the posting date or rating star, for example, trustpilot.com and reviews.io. amazon uses a crowdsourcing mechanism, a voting system, to gather feedback on review helpfulness, and then rank them based on the overall votes they received (amazon.com). a user can vote for a review as being helpful or unhelpful. amazon was estimated to receive a revenue of about $2.7 billion by providing simple question "was this review helpful to you?" @xcite . although such a voting system is helpful for customers, it has several limitations due to the inherent character of the voting process. there are number of reasons: 1) not all reviews get the helpfulness vote; 2) the helpfulness voting does not work for cold star review (i.e., a new user or a new review will have much less votes) @xcite ; 3) reviews receiving helpfulness votes would tend to gather more vote due to the snowball effect (e.g., phenomena such as social proof @xcite ). in this work we hypothesise that helpfulness of a review should be assessed based on three characteristics, namely relevance (whether the review discusses the key features relevant to a specific product), emotional intensity (level of emotions expressed within a review) and specificity (level of details discussed in a review). we motivate the importance of each of those features later in the paper. we then propose an unsupervised helpfulness ranking method that does not depend on the helpfulness votes and only takes under consideration the content of the review and the star rating. we demonstrate that our proposed method outperforms the state-of-the-art review ranking techniques, through an extensive empirical evaluation. the paper is organised as follows. in the next section we present an overview of the work that has been carried out in this space. following this, we provide the motivation and technical details of the proposed method. finally, the results of the experimental evaluation are demonstrated followed by the discussion. ## related work several approaches to automatically determining the helpfulness of online reviews have been explored in the past. in majority of the existing work, supervised machine learning models have been employed considering the problem as a predictive task (i.e. predict whether/how useful a review is) @xcite @xcite @xcite @xcite @xcite . with supervised approaches, various types of features such as linguistic features @xcite @xcite or textual features (i.e. polarity, subjectivity, entropy and readability) @xcite @xcite are first extracted from the reviews, with machine learning methods used over such data to train a predictive model. in a few papers, unsupervised learning based approaches have been used to rank reviews based on their helpfulness or relevance @xcite @xcite . it is very apparent that the majority of work has been focused on using supervised machine learning and unsupervised learning has not been well explored in this space. supervised learning methods depend on large, annotated datasets to train the model. unfortunately, most of the publicly available online reviews datasets do not have labels related to their helpfulness. this makes the unsupervised learning based approaches much more attractive and hence it is the focus of our work. a review ranking method based on unsupervised learning was proposed by @xcite . the authors first created a corpus of core dominant terms for the reviews representing the key aspects relevant to a specific product. dominant terms were obtained by computing the frequency of all terms in a reviews collection and re-ranking them by their frequency in the reference to the british national corpus, a baseline corpus. they named the corpus as virtual core (vc) review and represented it as feature vectors. following this, they ranked the reviews according to their distance from the virtual core review vector. they assumed that the smaller the distance between a review and the virtual core review, the more relevant/helpful the review is. @xcite proposed a ranking method to detect low quality reviews by using link analysis techniques. three ranking algorithms have been implemented in their study which are (1) pagerank algorithm @xcite , (2) hits algorithm @xcite , and (3) length algorithm. first, they construct a graph for each review of a product where the vertexes are sentences in a review. two directional edges between two vertexes are induced if they are similar according to specific pos tag i.e., nouns, adjectives, and verb. they compute the centrality scores of sentences using the pagerank and the hits algorithms. a score for each review was obtained by summing all the centrality scores of all the sentences in a review and then rank the review based on the high centrality scores. the length algorithm was used to rank all reviews based on total number of words. they count the number of words for each review and rank it based on the high score. the authors conjecture that high-quality review should contain more words than poor reviews. two baseline methods were used for comparison in their experimental evaluation. from the evaluation, it could however be observed that their results were only slightly inferior in comparison to the baselines. inspired by the work proposed in @xcite , @xcite developed mrr (most relevant review), a novel unsupervised algorithm to rank reviews based on their estimated relevance. mrr algorithm consists of three steps: (1) first, they construct a graph of reviews for each product where the nodes are the reviews and the edges are defined based on the similarity between pairs of reviews. two similarity scores are considered: cosine similarity between tf-idf vectors computed for each review, and similarity between rating scores of reviews (i.e., rating scores from 1 to 5 given by reviewers), (2) this is followed by graph pruning that works by removing all edges with the similarity scores lower that the minimum threshold value, (manually set as β=0.85533), (3) finally, the centrality scores are calculated for each review using pagerank algorithm. the authors hypothesise that the more central reviews should be considered as most relevant. two state-of-the-art unsupervised learning @xcite and two supervised learning methods (i.e., one of the method use the same features as @xcite ) were adopted in the experimental evaluation for comparison. although, their results were lower than those obtained by supervised learning methods, they outperformed the two unsupervised learning based approaches. in this work, we propose a new unsupervised method for ranking online reviews based on their helpfulness. apart from the relevance (as in case of the existing unsupervised techniques), our method also considers the emotional intensity and the specificity of the reviews while assessing their helpfulness; this makes it unlike any of the approaches discussed above. for the text representation, we apply the roberta state-of-the-art language model as opposed to tf-idf used by the existing unsupervised methods. ## methodology the key novelty of the proposed method is that it incorporates three different characteristics of online reviews while ranking them according to their helpfulness. we hypothesise that the helpfulness of a review should be determined based on the following features: a) relevance. relevance indicates how well a review matches with customer's specific information needs @xcite . in other words, a helpful review should discuss the key features of a product, which are important for the future buyers (e.g., "the camera is easy to use, it is compact and perfect for travelling."). review's relevance has been modelled by the existing work @xcite using graph composed of all reviews, their similarities and various centrality measures. it was assumed that the reviews that are the most central within the graph contain the most relevant information about the product. in our work, we take a similar approach, however, instead of graphs we used a simpler pair similarity based method. b) emotional intensity. we hypothesise that emotions play an important part in a review process as they allow customers to express their feelings and experiences through opinions. therefore, a good review should contain a good balance of both, facts and emotions. the relationship between helpfulness of online reviews and emotions have been explored by @xcite where they stud-ied which emotions are important for helpfulness prediction. @xcite used emotions to detect helpful reviews by applying different classification models (i.e., svm, random forest, and naïve bayes) and demonstrated that their approach outperformed methods using pos tagging features. emotion information has not been considered by any of the existing unsupervised methods. in this work, we propose to consider the level of emotions contained within a review as one of the factors in determining their helpfulness. c) specificity. a review of a product will be considered as useful/informative if it discusses various features of the products. in other words, instead of just expressing satisfaction/dissatisfaction from a product (e.g. "i hate this camera and would not recommend it"), it is much more helpful if the review explains what good or bad there is about the product (e.g. "the battery life is too short and the zoom is rather poor."). the greater number of different features is mentioned in a review, the more informative the review is for any potential buyer/customer. it should be noted that there is a distinct difference between the relevance and the specificity. with relevance, we assess whether the key characteristic of a product was discussed. while with specificity, we evaluate the level of details that was provided while discussing different features of a product. following this reasoning, we propose to consider the number of different entities mentioned in the reviews while ranking the reviews based on their helpfulness. such a specificity feature has also not been considered by any of the existing work. apart from the aforementioned characteristics, we also consider the star rating of the reviews in our ranking process. it has been demonstrated in the literature that the application of star rating is beneficial when evaluating the helpfulness of a review @xcite @xcite . the pseudocode of our proposed methods is presented in algorithm 1. the input to the method is a collection of reviews related to the same products. each review contains the review text (r) and the star rating associated with this review (s). in the first step of the algorithm, the input reviews are ranked separately on the basis of their relevance, emotional intensity and specificity. for the relevance ranking, we create a product-specific "summary document" (sum), which contains all individual reviews collated together. the summary document and each individual review are then converted into vectors using the roberta pre-trained language model @xcite . for this part, any other embedding model (such as word2vec or glove) can be considered. we used the roberta model as it has recently received state-of-the-art results on many nlp benchmark datasets @xcite . following this, the cosine similarity between each individual review and the summary document is calculated as its relevance score. it is worth noting that the proposed relevance ranking method is much simpler and faster than those of the baseline, which uses graphs to model similarity between reviews. with the second ranking, the reviews are ranked based on their emotional intensity. to identify different emotions in the reviews we used the depechemood++ @xcite lexicon that contains 187942 words with 8 emotions intensity value for each word; this could be replaced with any emotion lexicon. for each review, we first identify all words which are present in the lexicon. following this, all the intensity values assigned to those words in the lexicon are added together. the final emotion score assigned to each review is the accumulation of intensity value by summing all emotion words within this review. finally, for the specificity ranking, we first apply name entity recognition and extract entities from the reviews using the nltk library 2 . we calculate the specif icity score for each review as the sum of all entities that it contains. all the reviews are then sorted separately based on the three scores. as the outputs of the aforementioned steps, we obtained three rankings of the reviews, which were constructed based on the relevance, emotional intensity, and specificity of the reviews (lines 15-17). as mentioned earlier, we also consider the star rating in our ranking method as it is considered as an good indicator of reviews helpfulness @xcite @xcite . we process the star rating by calculating the absolute deviation. the use of star rating deviation as a feature has been demonstrated in @xcite @xcite @xcite 2 https://www.nltk.org/book/ch07.html @xcite and some of the authors apply absolute deviation for the star rating @xcite @xcite . first, we calculate the average of all star ratings of a product review (line 20). in the next step, for each review r i , we calculate its absolute deviation (ad) from the average star rating as per eq 1. the rad value will be further included in the final ranking process together with the other three rankings as explained below. for combining the three rankings (i.e., relevance, emotional intensity and specificity), we applied the z-score minimization method (standard score, 2021). first, the mean (µ) and the standard deviation (σ) of the three ranking positions are computed for each review r @xmath0. in the next step we calculate the z-score distance matrix calculating the z-score for each review and every possible ranking position according to the following formula (lines 21-26): ## result and discussion the results obtained by each of the evaluated methods on each of the four datasets are presented in tables 2 3 4 5 . each table demonstrates the result obtained by each of the methods with and without incorporating the star rating in the process. the first row in each of the tables refers to the results obtained by the state-of-the-art (mrr) unsupervised baseline @xcite . rows 2 and 3 show the results obtained by our method based only on the relevance ranking and using the full text or the summary of the reviews, respectively. the last two rows refer to the results obtained by the method when all three rankings were incorporated in the process. we evaluate our ranking quality using ndcg metrics and we take four different ranking positions. those are ndcg@3, ndcg@5, ndcg@7, and ndcg@10 where the number after the ndcg@ represent the number of reviews taken for evaluation from the top rank position. from the results presented in tables 2-5 we can observe that for each of the four datasets, the proposed method performed better when all three ranking were incorporated. this indicated that the emotional intensity and the specificity of a review are useful when determining its helpfulness. it can also be noted that our method obtained better results when the star rating is used when creating the final ranking. the difference is particularly apparent for the books dataset. finally, we can see that the proposed method performed better when applied with the full review content (relevance(full text)+emotion+specify) than with the summary (relevance(summary)+emotion+specify) with three out of four datasets. the only case when using the summaries of the reviews made a positive difference is the cds & vinyls dataset. looking at the overall results (both with and without rating star) we can conclude that our proposed method performs best when each of the three rankings is performed on the full reviews' content and when the rating star is considered. ## conclusion this paper addresses the problem of online reviews ranking according to their helpfulness. we propose an unsupervised method, which first ranks the reviews based on their relevance, emotional intensity and specificity and then combine them in order to obtain the final helpfulness ranking. the perfor-mance of the method on four datasets that were created for the purpose of this study was evaluated using the ndcg metric. it was demonstrated that the method outperformed the state-of-the-art unsupervised online review ranking method proposed in @xcite in every case. in the future, we want to improve our ranking system by applying different features and ranking method. some features such as linguistic features, positive and negative emotion, or topic sentences may be explore in the ranking system. moreover, different combination ranking method such as schulze @xcite or borda count @xcite or another ranking method could be explored to improve the performance.