Evaluating Content Features and Classification Methods for Helpfulness Prediction of Online Reviews: Establishing a Benchmark for P ortuguese (2022)

Abstract: Over the years, the review helpfulness prediction task has been the subject of several works, but remains being a challenging issue in Natural Language Processing, as results vary a lot depending on the domain, on the adopted features and on the chosen classification strategy. This paper attempts to evaluate the impact of content features and classification methods for two different domains. In particular, we run our experiments for a low resource language – Portuguese –, trying to establish a benchmark for this language. We show that simple features and classical classification methods are powerful for the task of helpfulness prediction, but are largely outperformed by a convolutional neural network-based solution.

## introduction the concern to facilitate users' decision-making is common in most e-commerce platforms. the possibility for customers to publicly provide product reviews is one of the consequences of this concern. this functionality allows future customers to read reviews from other customers and take their buying decision. despite being useful, the amount of generated data is very large, making it impossible for a human to read them all. moreover, a large part of this data can be considered unwanted, containing poorly written texts, vague opinions and texts of dubious quality @xcite , making it difficult to find relevant content. the helpfulness voting functionality that some e-commerce platforms adopt tries to address the above problem, ranking the reviews and showing the most helpful ones to the customers. however, manual voting has some drawbacks, as new helpful reviews take time to get enough votes and gain a visible position. the solution is to automatically predict the helpfulness of reviews. despite the usefulness of the task of helpfulness prediction and its practical implications, literature has shown that it is a challenging open issue in natural language processing (nlp). performance results vary drastically across domains and there are several different features and classification methods in the area, as discussed in @xcite . this paper aims to investigate such issues and to identify relevant features and methods for helpfulness prediction. we provide a qualitative and quantitative study of the impact of key content features in two different domains (apps and movies). by content features, we mean those that are related to the information that can be extracted directly from the review, such as the text and the "stars" given by the author. we also perform a comparative study of various classical and deep machine learning classifiers. we show that simple features and classical classification methods may be powerful for the task, but they are largely outperformed by a convolutional neural network-based approach, which reaches a f1-score of 0.90 for apps and 0.74 for movies. it is also relevant to cite that we run our experiments for a low resource language -brazilian portuguese -, bringing relevant contributions for nlp for portuguese and establishing a benchmark for the task. the rest of the paper is organized as follows. section 2 shows the main related work. in section 3, we describe the experimental setting adopted in this work. section 4 reports the achieved results and section 5 brings some final remarks. ## related work the main research line in review helpfulness prediction aims to predict the helpfulness score for a set of reviews. the helpfulness score is defined as shown in equation 1 and can be used as the target for regression, binary classification, or ranking. the score regression aims to predict the helpfulness score h ϵ [0, 1]. for binary classification, a threshold is applied in helpfulness score (e.g., @xmath0.5) and all reviews with a helpfulness score above the threshold are classified as helpful; otherwise, they are classified as not helpful. review ranking seeks to order the reviews by their helpfulness according to a reference ranking. in order to understand the helpfulness of online customer reviews, researches have performed several analyses. it is worth mentioning classical works like the ones of @xcite and @xcite that introduce many types of features for helpfulness prediction. @xcite split the features in 5 categories, all considered to be content features: structural, lexical, syntactic, semantic and meta-data features. they build a model for a regression task and a model for a ranking task using the svm algorithm. using a dataset of reviews on two products (mp3 players and digital cameras) extracted from amazon.com, the best results are achieved with the combination of length, unigram and number of stars features. in a similar way, @xcite propose three categories of features, also for a dataset extracted from amazon.com. their features include lexical similarity (cosine similarity over tf-idf vectors), shallow syntactic features (proper nouns, modal verbs, interjection, etc.) and lexical subjectivity clues (subjective adjectives, subjective nouns, etc.). the authors model two regressors using svr (support vector regression) and slr (simple linear regression) techniques, obtaining the best results by combining all the features. @xcite , in addition to the features already used by @xcite , propose the use of trigrams, comparison expressions ("compare to" or "adj + er than"), degree of detail and pros and cons. using an svm classifier, the authors address the helpfulness prediction task as a three-class classification: helpful positive reviews, helpful negative reviews, and unhelpful reviews. furthermore, by running a series of experiments with one less feature each time, they found that the "detail" feature is the most important one, followed by length, number of stars and unigram. more recently, researchers are using more robust methods for helpfulness prediction. it is the case of @xcite , that use bert (bidirectional encoder representations from transformers) @xcite along with the features of star rating and product type. with this combination, the authors model a neural network to predict the helpfulness score for reviews extracted from amazom.com. wang et al. (2020) also use bert, but the authors add more features (number of words, number of sentences, rating, etc.) than xu et al. ( 2020 ) and compare the bert-based approach to svm and cnn models. the neural network-based classifiers achieved similar results to svm using all features. @xcite propose the use of syntactic features along with bert sentence embeddings to helpfulness classification. the work compares some cnn models with bert and perform an ablation study with all syntactic features. their results showed high recall but very low precision values. in terms of f1-score, bert achieved the best results and the main feature was star rating. all these researches have in common the use of content features. the results of methods using handcrafted features were better or very close to state-of-the-art classifiers (using bert and cnn, for instance). in such setting, this paper aims at further exploring such issues, specially for the context of portuguese, a low resource language. we present our experiment setting in what follows. ## results we explored the following classical classification strategies in this work: naive bayes (nb), support vector machines (svm), decision tree (dt), random forest (rf), neural network multilayer perceptron (nn) and a dummy classifier. more sophisticated (deep) strategies that we tested are a bert-based classifier and a convolutional neural average sentence size in terms of words @xcite spacy with portuguese language model number of sentences (num-s) total of sentences in the review @xcite number of words (num-w) total of words in the review @xcite ; mudambi and schuff, 2010) star rating (star-r) the review-assigned product star rating (huang et al., 2015) -readability features (read) measure how easy a text is to read and include the following features: automated readability index (ari), coleman-liau index (cli), flesch reading ease (fre), flesch-kincaid grade level (fkgl), gunning fog index (gfi) and smog (dubay, 2004; ghose and ipeirotis, 2011) readability features based on (antunes and lopes, 2019) spelling errors (spell) number of misspelled words in review (ghose and ipeirotis, 2011) number of words not found in wiktionary 1 and unitex-pb lexicons (muniz, 2004) dominant terms (dom-terms) presence of important terms in reviews, considering their specificity for the domain @xcite we use the nilc corpus @xcite to calculate the frequencies of words that do not belong to the domains product aspects (prod-feat) presence of product aspects in the reviews @xcite @xcite we manually extract the features of texts from the corpus development set. ## final remarks this paper synthesized a series of experiments on predicting review helpfulness, showing some relevant learned lessons and contributions (in particular, for brazilian portuguese, which is considered a low resource language). however, a lot remains to be investigated. we highlight two issues that concern us the most at this time. firstly, the different performances for different domains (across different classification methods) keep intriguing us. this is a known behavior in the sentiment analysis area, and we corroborate it by testing new domains in this paper. we wonder whether new methods or features should be tested, maybe focusing on those that are more domain independent, or whether we should "transform" our data, "eliminating" domain specific traits. the other issue refers to the helpfulness predic-tion task itself. although the literature (including us) have exhaustively tried with this task, it is a highly subjective task that (indirectly) incorporate several other tasks, as subjectivity classifi-