Improving Document-Level Sentiment Analysis with User and Product Context (2020)

Abstract: Past work that improves document-level sentiment analysis by encoding user and product in- formation has been limited to considering only the text of the current review. We investigate incorporating additional review text available at the time of sentiment prediction that may prove meaningful for guiding prediction. Firstly, we incorporate all available historical review text belonging to the author of the review in question. Secondly, we investigate the inclusion of his- torical reviews associated with the current product (written by other users). We achieve this by explicitly storing representations of reviews written by the same user and about the same product and force the model to memorize all reviews for one particular user and product. Additionally, we drop the hierarchical architecture used in previous work to enable words in the text to directly attend to each other. Experiment results on IMDB, Yelp 2013 and Yelp 2014 datasets show improvement to state-of-the-art of more than 2 percentage points in the best case.

## introduction document-level sentiment analysis aims to predict sentiment polarity of text that often takes the form of product or service reviews. @xcite demonstrated that modelling the individual who has written the review, as well as the product being reviewed, is worthwhile for polarity prediction, and this has led to exploratory work on how best to combine review text with user/product information in a neural architecture @xcite @xcite @xcite . a feature common amongst past studies is that user and product ids are modelled as embedding vectors whose parameters are learned during training. we take this idea a step further and represent users and products using the text of all the reviews belonging to a single user or product -see fig. 1 ## methodology an overview of our model architecture is shown in figure 1 (right). the input to our model consists of d, u, p, which are the document, the user id and the product id respectively. u and p are both mapped to embedding vectors, e u , e @xmath0. d is fed into the bert encoder to generate a document representation h @xmath1. we then inject e u and e p , to get the user-product biased document representation h @xmath2. finally, we feed the biased document representation h biased into a linear layer followed by a softmax layer to get the distribution of the sentiment label y. we use cross-entropy to calculate the loss between the predictions and ground-truth labels. injecting user and product preferences we adopt stacked multi-head-attention (q, k, v ) @xcite to model the connections between the current document and user/product vectors, which in this work correspond to all historical reviews composed by the user or about the product to date. in a typical dot-product attention (q, e u and e p are regarded as queries, h d as keys and values. we compute the user-specific document representation, c t u , and product-specific document representation, c t p as follows: we adopt a gating mechanism to obtain importance vectors, z u and z p , to control the contribution of user-specific and product-specific document representations to the output classification: finally, we obtain the biased document representation h biased by: updating the user and product matrix to implement our idea of using all reviews composed by u and all reviews about p, we incrementally add the current user/product-specific document representation to the corresponding entries in the embedding matrix at each step during training: ## conclusion in this paper, we propose a neural sentiment analysis architecture that explicitly utilizes all past reviews from a given user or product to improve sentiment polarity classification on the document level. our experimental results on the imdb, yelp-13 and yelp-14 datasets demonstrate that incorporating this additional context is effective, particularly for the yelp datasets. the code used to run the experiments is available for use by the research community. 4