Can You Distinguish Truthful from Fake Reviews? User Analysis and Assistance Tool for Fake Review Detection (2021)

Abstract: Customer reviews are useful in providing an indirect, secondhand experience of a product. People often use reviews written by other customers as a guideline prior to purchasing a product. Such behavior signifies the authenticity of reviews in e-commerce platforms. However, fake reviews are increasingly becoming a hassle for both consumers and product owners. To address this issue, we propose You Only Need Gold (YONG), an essential information mining tool for detecting fake reviews and augmenting user discretion. Our experimental results show the poor human performance on fake review detection, substantially improved user capability given our tool, and the ultimate need for user reliance on the tool.

## introduction the increasing prominence of e-commerce platforms gave rise to numerous customer-written reviews. the reviews, given their authenticity, provide important secondhand experience to other potential customers or to information-seeking functions such as search and question answering. meanwhile, fake reviews are increasingly becoming a social problem in e-commerce platforms @xcite @xcite . such deceptive reviews are either incentivized by the beneficiaries (i.e., sellers, marketers) or motivated by those with malicious intention to damage the reputation of the target product. to date, there have been many studies @xcite @xcite @xcite that address fake review detection in the field of natural language processing (nlp). the use of highperformance deep neural networks such as bert * equal contribution. figure 1 : yong -a prototype interface. given the review input by user, yong shows 1) whether it is gold or fake with 2) the probability (%), and 3) evidence that shows how much each word contributes to model's final decision. the highlighted evidences show the top p% proportion of the contributors, where the proportion can be adjusted using the horizontal slider bar. @xcite , fast and scalable anomaly detection algorithms like densealert @xcite have made effective and promising contributions in detection of fraudulent reviews. despite such contributions, these approaches only focus on better modeling to improve the accuracy in fake review detection, instead of its practical applications such as assisting users to distinguish fake reviews (i.e., an assistance tool) or filtering out deceptive texts for review-based question answering (qa) @xcite . in the line of human-computer interaction (hci), there have been a variety of studies on customer reviews @xcite @xcite . while there are gold reviews, which are authentic, real-user written reviews, there are fake, deceptive reviews as well. all of the previous works on review visualization and interaction implicitly assume the authenticity of collected reviews. furthermore, the previous works mentioned above confine the scope of research on reviews to interaction and visualization. we claim that user capabilities of distinguishing fake reviews are seriously unreliable as suggested in @xcite , and this motivates our work as practical research for helping humans discern fake reviews. while the two lines of research in nlp and hci have focused on improving the fake review detection models and developing effective visualization of reviews, respectively, the actual victims (the users) of fake reviews are being neglected. the challenges of carefully curated deceptive reviews on the web necessitate the need for an assistance tool that helps users avoid fraudulent information. in this work, we propose you only need gold (yong) (figure 1 ), a simple assistance tool that augments user discretion in fake review detection. yong is built upon the body of previous studies by fine-tuning bert and providing a self-explanatory gold indicator to assist users in exploring customer reviews. through a series of user evaluations, we reveal the over-confident nature of people despite their poor performance in distinguishing fake reviews from real ones and the need to implement an explainable, human-understandable features to guide user decisions in fake review detection. we also demonstrate that the application of yong effectively augments user performance. our contributions is two-fold, the tool and an extensive user understanding, as follows: ## method we build a tool that provides straightforward, intuitive features to guide user decision in fake review detection. the tool is built upon a state-of-the-art nlp model fine-tuned on the opspam data set. ## experiment and result 4.1 research question to validate the usefulness of our tool and further the understanding of users in the application of our tool, we define the following research questions (rq): rq1. "how do humans fare against machines on fake review detection?" rq2. "does yong augment human performance on the task?" rq3. "can we increase the level of human trust on yong by injecting prior knowledge about human and model performance?" rq4. "how much influence does each feature of the gold indicator have on human trust?" here, we define the term "trust" as the level of human reliance on the tool's decision. to be specific, as we evaluate in section 4.5, the level of trust is calculated on the participant-machine agreement (regardless of the ground-truth label). through the experiments that correspond to the rqs, we build a concrete understanding of user behaviors in the presence of customer-generated reviews and the gold indicator through human performance evaluation on fake review detection. ## discussion there are three major findings in this work. in a fake review detection setting, (i) human capability is unreliable and needs machine assistance, (ii) the interpretable hidden weights are hardly explicable, and (ii) for dnn-powered assistive tools like yong, it is essential to provide faith-gaining features for its users. a notable finding is the difference between interpretability and explainability. in this work, we distinguish the two terms to ease their equivocal use. interpretability refers to the property of being able to describe the cause and effect of the model through observation, whereas explainability refers to the human-understandable qualities that can be accepted on the terms of human logic and intuition. although numerous existing works @xcite @xcite have repeatedly provided an interpretable look into the model's decision making process through layerwise hidden vector or attention weight visualization, most stop at showing the colored vectors and matrices. in figure 2 , we can see how the model places much attention on proper and common nouns like "chicago," and "morning," that fail to disambiguate and explain the reasoning process of our model for a human to understand. we can also observe in ## conclusion our work proposed you only need gold (yong), an assistant tool for fake review detection. from a series of experiments, we deepened our understanding of human capability in fake review detection. we observed that people were generally overconfident with their ability to discern fake reviews from real ones, and we discovered that the model far outperforms its human counterparts, suggesting the need for effective design to convince users to trust the model decision. furthermore, our work reveals the need to develop more "explainable" tools and promotes collaboration of users and the machine for fake review detection. for future work, expanding the scope of our tool to other fields such as products and restaurants would likely contribute to its generalizability.