Item-based Collaborative Filtering with BERT (2020)

Abstract: In e-commerce, recommender systems have become an indispensable part of helping users explore the available inventory. In this work, we present a novel approach for item-based collaborative filtering, by leveraging BERT to understand items, and score relevancy between different items. Our proposed method could address problems that plague traditional recommender systems such as cold start, and “more of the same” recommended content. We conducted experiments on a large-scale real-world dataset with full cold-start scenario, and the proposed approach significantly outperforms the popular Bi-LSTM model.

## introduction recommender systems are an integral part of ecommerce platforms, helping users pick out items of interest from large inventories at scale. traditional recommendation algorithms can be divided into two types: collaborative filtering-based @xcite and content-based @xcite . however, these have their own limitations when applied directly to real-world ecommerce platforms. for example, traditional userbased collaborative filtering recommendation algorithms (see, e.g., @xcite find the most similar users based on the seed user's rated items, and then recommend new items which other users rated highly. for item-based collaborative filtering (see, e.g., @xcite , given a seed item, recommended items are chosen to have most similar user feedback. however, for highly active e-commerce platforms with large and constantly changing inventory, both approaches are severely impacted by data sparsity in the user-item interaction matrix. content-based recommendation algorithms calculate similarities in content between candidate items and seed items that the user has provided feedback for (which may be implicit e.g. clicking, or explicit e.g. rating), and then select the most similar items to recommend. although less impacted by data sparsity, due to their reliance on content rather than behavior, they can struggle to provide novel recommendations which may activate the user's latent interests, a highly desirable quality for recommender systems @xcite . due to the recent success of neural networks in multiple ai domains @xcite and their superior modeling capacity, a number of research efforts have explored new recommendation algorithms based on deep learning (see, e.g., @xcite @xcite . in this paper, we propose a novel approach for item-based collaborative filtering, by leveraging the bert model @xcite to understand item titles and model relevance between different items. we adapt the masked language modelling and next sentence prediction tasks from the natural language context to the e-commerce context. the contributions of this work are summarized as follows: ## item-based collaborative filtering with bert as mentioned earlier, for a dynamic e-commerce platform, items enter and leave the market continuously, resulting in an extremely sparse user-item interaction matrix. in addition to the challenge of long-tail recommendations, this also requires the recommender system to be continuously retrained and redeployed in order to accommodate newly listed items. to address these issues, in our proposed approach, instead of representing each item with a unique identifier, we choose to represent each item with its title tokens, which are further mapped to a continuous vector representation. by doing so, essentially two items with the same title would be treated as the same, and can aggregate user behaviors accordingly. for a newly listed item in the cold-start setting, the model can utilize the similarity of the item title to ones observed before to find relevant recommended items. the goal of item-based collaborative filtering is to score the relevance between two items, and for a seed item, the top scored items would be recommended as a result. our model is based on bert @xcite . rather than the traditional rnn / cnn structure, bert adopts transformer encoder as a language model, and use attention mechanism to calculate the relationship between input and output. the training of bert model can be divided into two parts: masked language model and next sentence prediction. we re-purpose these tasks for the e-commerce context into masked language model on item titles, and next purchase prediction. since the distribution of item title tokens differs drastically from words in natural language which the original bert model is trained on, retraining the masked language model allows better understanding of item information for our use-case. next purchase prediction can directly be used as the relevance scoring function for our item collaborative filtering task. ## results the results of our evaluation are presented in table. 1. we do not consider the traditional item-toitem collaborative filtering model @xcite here since the evaluation is conducted assuming a complete cold-start setting, with all seed items unobserved in the training set, resulting in complete failure of such a model. following the same reason, other approaches relying on unique item identifier (e.g. itemid) couldn't be considered either in our experiment. we believe its a practical experiment setting, as for a large-scale e-commerce platform, a massive amount of new items would be created every moment, and ignoring those items from the recommender system would be costly and inefficient. we observe that the proposed bert model greatly outperforms the lstm-based model. when only fine-tuned for the next purchase prediction task, our model exceeds the baseline by 310.9%, 96.6%, 93.9%, and 150.3% in precision@1, precision@10, recall@10, and ndcg@10 respectively. when fine tuning for the masked language model task is added, we see the metrics improved further by another 111.0%, 38.6%, 38.3%, and 64.0%. from the experiment, the superiority of proposed bert model for item-based collaborative filtering is clear. it is also clear that adapting the token distribution for the e-commerce context with masked language model within bert is essential for achieving the best performance. in order to visually examine the quality of recommendations, we present the recommended items for two different seed items in table . 2. for the first seed item 'marvel spiderman t-shirt small black tee superhero comic book character', most of the recommended items are t-shirts, paired with clothing accessories and tableware decoration, all having marvel as the theme. for the second seed item 'microsoft surface pro 4 12.3" multi-touch tablet (intel i5, 128gb) + keyboard', the recommended items span a wide range of categories including tablets, digital memberships, electronic accessories, and computer hardware. from these two examples, we see that the proposed model appears to automatically find relevant selection criteria without manual specification, as well as make decisions between focusing on a specific category and catering to a wide range of inventory by learning from the data. ## summary in this paper, we adapt the bert model for the task of item-based recommendations. instead of directly representing an item with a unique identifier, we use the item's title tokens as content, along with token embeddings, to address the cold start problem. we demonstrate the superiority of our model over a traditional neural network model in understanding item titles and learning relationships between items across vast inventory.