Multi-perspective Coherent Reasoning for Helpfulness Prediction of Multimodal Reviews (2021)

Abstract: As more and more product reviews are posted in both text and images, Multimodal Review Analysis (MRA) becomes an attractive research topic. Among the existing review analysis tasks, helpfulness prediction on review text has become predominant due to its importance for e-commerce platforms and online shops, i.e. helping customers quickly acquire useful product information. This paper proposes a new task Multimodal Review Helpfulness Prediction (MRHP) aiming to analyze the review helpfulness from text and visual modalities. Meanwhile, a novel Multi-perspective Coherent Reasoning method (MCR) is proposed to solve the MRHP task, which conducts joint reasoning over texts and images from both the product and the review, and aggregates the signals to predict the review helpfulness. Concretely, we first propose a product-review coherent reasoning module to measure the intra- and inter-modal coherence between the target product and the review. In addition, we also devise an intra-review coherent reasoning module to identify the coherence between the text content and images of the review, which is a piece of strong evidence for review helpfulness prediction. To evaluate the effectiveness of MCR, we present two newly collected multimodal review datasets as benchmark evaluation resources for the MRHP task. Experimental results show that our MCR method can lead to a performance increase of up to 8.5% as compared to the best performing text-only model. The source code and datasets can be obtained from https://github.com/jhliu17/MCR .

## introduction product reviews are essential information sources for consumers to acquire useful information and make purchase decisions. many e-commerce sites such as amazon.com offer reviewing functions that encourage consumers to share their opinions and experiences. however, the user-generated reviews vary a lot in their qualities, and we are continuously bombarded with ever-growing, noise information. therefore, it is critical to examine the quality of reviews and present consumers with useful reviews. motivated by the demand of gleaning insights from such valuable data, review helpfulness prediction has gained increasing interest from both academia and industry communities. earlier review helpfulness prediction methods rely on a wide range of handcrafted features, such as semantic features @xcite , lexical features @xcite , and argument based features @xcite , to train a classifier. the success of these methods generally relies heavily on feature engineering which is labor-intensive and highlights the weakness of conventional machine learning methods. in recent years, deep neural networks such as cnn @xcite and lstm @xcite have become dominant in the literature due to their powerful performance for helpfulness prediction by learning text representation automatically. note that these existing works on review helpfulness prediction mainly focus on the pure textual data. as multimodal data become increasingly popular in online reviews, multimodal review analysis (mra) has become a valuable research direction. in this paper, we propose the multimodal review helpfulness prediction (mrhp) task which aims at exploring multimodal clues that often convey comprehensive information for review helpfulness prediction. in particular, for the multimodal reviews, the helpfulness of reviews is not only determined by the textual content but rather the combined expression (e.g., coherence) of multimodality data (e.g., texts and images). taking the reviews in in this paper, we explore both text and images in product reviews to improve the performance of review helpfulness prediction. we design a novel multi-perspective coherent reasoning method (denoted as mcr) to tackle the mrhp task. concretely, we propose a product-review coherent reasoning module to effectively capture the intra-and inter-modal coherence between the target product and the review. in addition, we also devise an intrareview coherent reasoning module to capture the coherence between the text content and images of the review, which is a piece of strong evidence for review helpfulness prediction. finally, we formulate the helpfulness prediction as a ranking problem and employ a pairwise ranking objective to optimize the whole model. we summarize our main contributions as follows. (1) to the best of our knowledge, this is the first attempt to explore both text and images in reviews for helpfulness prediction, which is defined as the mrhp task. (2) we propose a multi-perspective coherent reasoning method for the mrhp task to conduct joint reasoning over texts and images from both the product and the review, and aggregate the signals to predict the helpfulness of multimodal reviews. (3) we present two newly-collected multimodal review datasets for helpfulness prediction of multimodal reviews. to facilitate research in this area, we will release the datasets and source code proposed in this paper, which would push forward the research in this field. (4) extensive experiments on two collected datasets demonstrate that our mcr method significantly outperforms other methods. ## related work most conventional approaches on review helpfulness prediction focus solely on the text of reviews, which can be generally divided into two categories based on the way of extracting predictive features: machine learning based methods with hand-crafted features @xcite product information teflon pans 1 set of 3 pcs 1042-non-stick set of 3 review 1 (helpfulness score: 2) overall, it is quite satisfactory. thanks to the seller. review 2 (helpfulness score: 4) for that price, it is more than satisfactory, even though there are a few scratches in the pan and the small frying pan, the package is very neat, the frying pan has been used as if it's a little burnt, it looks like it can't stand the heat, but overall i like it. review 3 (helpfulness score: 0) recommend for the price. yes, the package is neat but the pan has scratched. it is unfortunate for the delivery. i ordered 4 items in this shop. but the postage has to pay double and quite very expensive. subsequently, several works have been proposed to explore not only the reviews but also the users and target products for helpfulness prediction of reviews. @xcite argued that the helpfulness of a review should be aware of the meta-data (e.g., title, brand, category, description) of the target product besides the textual content of the review itself. to this end, a deep neural architecture was proposed to capture the intrinsic relationship between the meta-data of a product and its numerous reviews. @xcite proposed to leverage the reviews, the users, and items together for helpfulness prediction of reviews and devised a categoryaware graph neural networks with one shared and many item-specific graph convolutions to learn the common features and each item's specific criterion for helpfulness prediction. different from the above methods, we take full advantage of the text content and images of reviews by proposing a novel hierarchical coherent reasoning method to learn the coherence between text content and images in a review and the coherence between the target product and the review. ## methodology the overall architecture of our mcr method is illustrated in figure 1 . our multi-perspective coherent reasoning consists of two perspectives of coherence: (i) the intra-and inter-modal coherence between a review and the target product and (ii) the intra-review coherence between the text content and images in the review. in the following sections, we will provide the problem definition of review helpfulness prediction and introduce each component of our mcr model in detail. ## conclusion multimodal review analysis (mra) is extremely important for helping businesses and consumers quickly acquire valuable information from usergenerated reviews. this paper is the first attempt to explore the multimodal review helpfulness prediction (mrhp) task, which aims at analyzing the review helpfulness from text and images. we propose a multi-perspective coherent reasoning (mcr) method to solve mrhp task, which fully explores the product-review coherence and intra-review coherence from both textual and visual modalities. in addition, we construct two multimodal review datasets to evaluate the effectiveness of mcr, which may push forward the research in this field. extensive experimental results demonstrate that mcr significantly outperforms baselines by comprehensively exploiting the images associated with the reviews.