Exploiting Rich Textual User-Product Context for Improving Personalized Sentiment Analysis (2023)

Abstract: User and product information associated with a review is useful for sentiment polarity prediction. Typical approaches incorporating such information focus on modeling users and products as implicitly learned representation vectors. Most do not exploit the potential of historical reviews, or those that currently do require unnecessary modifications to model architectureor do not make full use of user/product associations. The contribution of this work is twofold: i) a method to explicitly employ historical reviews belonging to the same user/product in initializing representations, and ii) efficient incorporation of textual associations between users and products via a user-product cross-context module. Experiments on the IMDb, Yelp-2013 and Yelp-2014 English benchmarks with BERT, SpanBERT and Longformer pretrained language models show that our approach substantially outperforms previous state-of-the-art.

## introduction it has been repeatedly shown that the user and product information associated with reviews is helpful for sentiment polarity prediction @xcite @xcite . just as the same user is expected to have consistent narrative style and vocabulary, the reviews belonging to the same product are expected to exhibit similar vocabulary for specific terms. most previous work models user and product identities as representation vectors which are implicitly learned during the training process and only focus on the interactions between either the user or product and the review text @xcite @xcite @xcite . this brings with it two major shortcomings: i) the associations between users and products are not fully exploited, and, ii) the text of historical reviews is not used. to tackle the first shortcoming, amplayo et al. @xcite propose to incorporate similar user and product representations for review sentiment classifica- it 's definitely a place for a cool date…… i wouldn't do dinner here again…… this place has a really chill , almost a hipster feel…… ……………….. the food is the reason for the rating…… it packs heats, but for me its lacking in the heat…… the food were fine but absolutely nothing special ……………….. what kind of users like this product? what kind of products this user like? similar users similar products historical reviews historical reviews tion. however, their approach ignores the associations between users and products. to tackle the second shortcoming, @xcite propose to explicitly use historical reviews in the training process. however, their approach needs to incrementally store review representations during the training process, which results in a more complex model architecture, where the magnitude of the user and product matrix is difficult to control when the number of reviews grow very large. as shown in figure 1 , we propose two simple strategies to address the aforementioned issues. firstly, we use pre-trained language models (plms) to pre-compute the representations of all historical reviews belonging to the same user or product. historical review representations are then used to initialize user (or product) representations by average pooling over all tokens before again average pooling over all reviews. this allows historical review text to inform the user and product preference, which we believe is potentially more advantageous than implicitly learned representations. time and memory costs are minimized compared to @xcite since the representations of historical reviews are average pooled and the precomputation is one-time. secondly, we propose a user-product cross-context module which interacts on four dimensions: user-to-user, product-to-product, user-to-product and product-to-user. the former two are used to obtain similar user (or product) information, which is useful when a user (or product) has limited reviews. the latter two are used to model the product preference of the user (what kind of products do they like and what kind of ratings would they give to similar products?) and user preference associated with a product (what kinds of users like such products and what kinds of ratings would they give to this product?). we test our approach on three benchmark english datasets -imdb, yelp-2013, yelp-2014. our approach yields consistent improvements across several plms (bert, spanbert, longformer) and achieves substantial improvements over the previous state-of-the-art. ## methodology an overview of our approach is shown in figure 2 . we firstly feed the review text, d, into a plm encoder to obtain its representation, h d . h d is then fed into a user-product cross-context module consisting of multiple attention functions together with the corresponding user embedding and product embedding. the output is used to obtain the distribution over all sentiment labels. the architecture design is novel in two ways: 1) the user and product embedding matrices are initialized using representations of historical reviews of the corresponding users/products, 2) a user-product crosscontext module works in conjunction with 1) to model textual associations between users and products. ## conclusion and future work in order to make the best use of user and product information in sentiment classification, we propose a text-driven approach: 1) explicitly utilizing historical reviews to initialize user and product representations 2) modeling associations between users and products with an additional user-product crosscontext module. the experiments conducted on three english benchmark datasets -imdb, yelp-2013 and yelp-2014 -demonstrate that our approach substantially outperforms previous stateof-the-art approaches and is effective for several plms. for future work, we aim to apply our approach to more tasks where there is a need to learn representations for various types of attributes, and to explore other compositionality methods for generating user/product representations.