BERT -Based Neural Collaborative Filtering and Fixed-Length Contiguous Tokens Explanation (2020)

Abstract: We propose a novel, accurate, and explainable recommender model (BENEFICT) that addresses two drawbacks that most review-based recommender systems face. First is their utilization of traditional word embeddings that could influence prediction performance due to their inability to model the word semantics’ dynamic characteristic. Second is their black-box nature that makes the explanations behind every prediction obscure. Our model uniquely integrates three key elements: BERT, multilayer perceptron, and maximum subarray problem to derive contextualized review features, model user-item interactions, and generate explanations, respectively. Our experiments show that BENEFICT consistently outperforms other state-of-the-art models by an average improvement gain of nearly 7%. Based on the human judges’ assessment, the BENEFICT-produced explanations can capture the essence of the customer’s preference and help future customers make purchasing decisions. To the best of our knowledge, our model is one of the first recommender models to utilize BERT for neural collaborative filtering.

## introduction in recommender systems research, collaborative filtering (cf) is the dominant state-of-the-art recommendation model, which primarily focuses on learning accurate representations of users (user preferences) and items (item characteristics) @xcite . the earliest recommender models learned these representations based on user-given numeric ratings that each item received @xcite . however, ratings, which are values on a single discrete scale, oversimplify user preferences and item characteristics @xcite . the large amount of users and items in a typical online platform consequently results in a highly sparse rating matrix, making it hard to learn accurate representations @xcite . to alleviate these issues, review texts have instead been utilized to model such representations for subsequent recommendation and rating prediction, and this approach has attracted growing attention in research @xcite . the main advantage of reviews as the source of features is that they can cover user opinions' multi-faceted substance. because users can explain their reasons underlying their given ratings, reviews contain a large amount of latent information that is both rich and valuable, and that cannot be otherwise obtained from ratings alone @xcite . recently, models that incorporate user reviews have yielded state-of-the-art performances @xcite . these approaches learn user and item representations by using traditional word embeddings (e.g., word2vec, glove) to map each word in the review into its corresponding vector. the review is transformed into an embedded matrix before being fed to a convolutional neural network (cnn) @xcite . cnns have been shown to effectively model reviews and have illustrated outstanding results in numerous natural language processing tasks @xcite . nevertheless, there are drawbacks that most review-based recommender models experience. first is the utilization of traditional or mainstream word embeddings to learn review features. their static nature is a hindrance, as each word sense is associated with the same embedding regardless of the context. in other words, such embeddings cannot identify the dynamic nature of each word's semantics. for review-based recommenders, this could be an issue in modeling users and items, which could, in turn, affect recommendation performance @xcite . also, once a cnn is fed with the matrix of word embeddings, the word frequency information of contextual fea-tures, said to be crucial for modeling reviews, will be lost @xcite . another drawback is the inherent black-box nature of deep learning-based models that makes the explanations behind every prediction obscure @xcite . the complex architecture of hidden layers has opaqued the models' internal decision-making processes @xcite . providing explanations could help persuade users to make decisions and develop trust in a recommender system @xcite @xcite . however, this leads us to a dilemma, i.e., a trade-off between accuracy and explainability. usually, the most accurate models are inherently complicated, non-transparent, and unexplainable @xcite . the same is also true for explainable and straightforward methods that sacrifice accuracy. formulating models that are both explainable and accurate is a challenging yet critical research agenda for the machine learning community to ensure that we derive benefits from machine learning fairly and responsibly @xcite . in this paper, we propose a unique model: bert-based neural collaborative filtering and fixed-length contiguous tokens explanation (benefict). our model learns user and item representations simultaneously using two parallel networks. to address the first drawback, we incorporate bert as a key component in each parallel network. bert affords us to extract more meaningful, contextualized features adaptable to arbitrary contexts; such features cannot be derived from mainstream word embeddings @xcite . bert can also retain the word frequency information that makes cnn an unnecessary component of our model. once user and item representations are learned, they are concatenated together in a shared hidden space before being finally fed to an optimal stack of multilayer perceptron (mlp) layers that serve as benefict's interaction function. to address the second drawback, we introduce a novel component in our model that integrates bert's self-attention and an implementation of the fixed-length maximum subarray problem (msp), which is considered to be a classic computer science problem. bert applies self-attention in each encoder layer that consequently produces selfattention weights for each token. these are passed to the successive encoder layers through feedforward networks. we argue that these self-attention weights can be the basis for explaining rating predictions. based on this premise, msp then selects a segment or subarray of consecutive tokens that has the maximum possible sum of self-attention weights. ## related work and concepts designing a cf model involves two crucial steps: learning user and item representations and modeling user-item interactions based on those representations @xcite . before the advancements provided by neural networks, matrix factorization (mf) was the dominant model representing users and items as vectors of latent factors (called embeddings) and models user-item interactions using the inner product operation. the said operation leads to poor performance because it is sub-optimal for learning rich yet complicated patterns from realworld data @xcite . to address this scenario, neural networks (nn) have been integrated into recommender architectures. one of the initial works that have laid the foundation in employing nn for cf is ncf @xcite . their framework, originally implemented for rating-based, implicit cf, learns non-linear interactions between users and items by employing mlp layers as their interaction function, granting it a high degree of non-linearity and flexibility to learn meaningful interactions. two common designs have emerged when it comes to leveraging mlp layers: placing an mlp above either the concatenated user-item embeddings @xcite or the element-wise product of user and item embeddings @xcite . as far as rating prediction is concerned, two notable recommender models have yielded significant state-of-the-art prediction performances. deepconn is the first deep model that represents users and items from reviews jointly @xcite . it consists of two parallel, cnn-powered networks. one network learns user behavior by examining all reviews that he has written, and the other network models item properties by exploring all reviews that it has received. a shared layer connects these two networks, and factorization machines capture user-item interactions. the second model is narre, which shares certain similarities with deepconn. narre is also composed of two parallel networks for user and item modeling with respective cnns to process reviews @xcite . rather than concatenating reviews to one long sequence the same way that deepconn does, their model introduces an attention mechanism that learns review-level usefulness in the form of attention weights. these weights are integrated into user and item representations to enhance the embedding quality and the subsequent prediction accuracy. both deepconn and narre employ traditional word embeddings. other relevant studies have claimed to provide explanations for recommendations such as efm @xcite , scvr @xcite , and trirank @xcite . these models initially extract aspects and opinions by performing phraselevel sentiment analysis on reviews. afterward, they generate feature-level explanations according to product features that correspond to user interests @xcite . however, these models have some limitations; manual preprocessing is required for sentiment analysis and feature extraction, and the explanations are simple extraction of words or phrases from the review text @xcite . this also has the unintended effect of distorting the reviews' original meaning @xcite . another limitation is that textual similarity is solely based on lexical similarity; this implies that semantic meaning is ignored @xcite . ## methodology benefict, as illustrated in figure 1 , has two parallel networks to model user and item embeddings that both utilize bert. hereafter, we will only illustrate the user modeling process because the same is also observed for item modeling, with their inputs as the only difference. ## conclusion and future work we have successfully implemented a novel recommender model that uniquely integrates bert, mlp, and msp. benefict's predictive capability is validated by experiments performed on amazon and yelp datasets, consistently outperforming other state-of-the-art models. moreover, its explanation generation capability is verified by human judges. we argue that our work offers an avenue to help bridge the research gap between accuracy and explainability. in the future, we will consider incorporating other neural components, such as attention mechanisms, in improving the user-item modeling process. we also intend to enhance the expressiveness and the overall quality of the generated explanations.