PUNR : Pre-training with User Behavior Modeling for News Recommendation (2023)

Abstract: News recommendation aims to predict click behaviors based on user behaviors. How to effectively model the user representations is the key to recommending preferred news. Existing works are mostly focused on improvements in the supervised fine-tuning stage. However, there is still a lack of PLM-based unsupervised pre-training methods optimized for user representations. In this work, we propose an unsupervised pre-training paradigm with two tasks, i.e. user behavior masking and user behavior generation, both towards effective user behavior modeling. Firstly, we introduce the user behavior masking pre-training task to recover the masked user behaviors based on their contextual behaviors. In this way, the model could capture a much stronger and more comprehensive user news reading pattern. Besides, we incorporate a novel auxiliary user behavior generation pre-training task to enhance the user representation vector derived from the user encoder. We use the above pre-trained user modeling encoder to obtain news and user representations in downstream fine-tuning. Evaluations on the real-world news benchmark show significant performance improvements over existing baselines.

## introduction news recommendation is an essential technology for predicting candidate news via the history of user behaviors. it is widely used in online news feeds and platforms to model user behaviors and push personalized news to their users. existing news recommendation methods often employ pre-trained language models (plms) to produce news vectors and user vectors, and match personalized candidate news based on the dot product distances of these two vectors. for example, nrms-bert @xcite replaces multi-head self-attention with pre-trained plm, such as bert @xcite , roberta @xcite and unilm @xcite , to figure 1 : the architecture of our user modeling method for new recommendations in downstream fine-tuning. a siamese encoder is used for both modeling the news and user embeddings. note that we use the textual titles of user-clicked news as inputs in pre-training, rather than a pure news id. on the contrary, unbert @xcite ) proposes to concatenate the candidate news and user behavior news together, and predict the matching score via a classification head with the wordlevel vector and the news-level vector. amm @xcite concatenates the cross pairs of title, abstract, and body together between the users' history news and candidate news, to extract multifield matching representation in both within-field and cross-field. such single-tower classification techniques result in good user-candidate behavior modeling but are relatively computationally expensive. in order to achieve joint modeling of user behaviors while maintaining a balanced computational offload, as is shown in figure 1 , we simplify the architecture of the user encoder and only use a siamese encoder for modeling both news vectors and user vectors. the concatenation of user history news brings better alignment of user behaviors. and we refer to this joint modeling of users' clicked histories as user modeling. effectively recommending content to users relies heavily on the ability to accurately model their behaviors @xcite . ptum @xcite proposes pre-training multi-head self-attention layers from scratch to predict masked behaviors and the next k behaviors. however, ptum only utilizes a small amount of in-domain data and does not leverage the power of plms. unfortunately, pretraining for news recommendation has not yet been extensively explored, as most existing methods focus on improving the fine-tuning stage by adapting an off-the-shelf plm as the news encoder. we argue that pre-training a plm specifically tailored for news recommendation is essential for achieving optimal performance. moreover, research on modeling user behaviors for news recommendations is still in its early stages. in our work, we propose punr, pre-training with user behavior modeling for news recommendation, seeking to incorporate unsupervised pre-training signals for better user modeling ability. on top of the siamese architecture, we incorporate two pre-training tasks tailored for user modeling. detailed designs are shown in figure 2 . inspired by recent representation generative pretraining technology in ir community @xcite , the information can be compressed into the representation vector through a generative pre-training task with a shallow transformer decoder. this auto-regression pre-training schema enlights us to explore new pre-training paradigms for better modeling a user vector in news recommendations. to the best of our knowledge, we make the first effort to bring unsupervised pre-training tasks to the plm-based news recommender system. the joint use of user behavior generation and user behavior masking design for user vector optimization makes a steady recommendation improvement. we also propose a simple yet effective siamese architecture for encoding both the news vectors and user vectors. experiment results on the real-world news recommendation task show considerable performance gains over multiple baselines. in addition, we conduct several ablation studies to verify the robustness of our method. our contribution can be summarized as follows: • we propose an effective user modeling siamese encoder architecture for both encoding the news and user representations. token embeddings positional embeddings news segment embeddings user history news cls masked user embeddings user embeddings … + + + + + + + + + + + + + + + + + + [m] mentations of the pre-training and fine-tuning. ## experiments in this section, we introduce the experimental settings of our work, including the pre-training, finetuning, and baseline methods. then we analyze the main experimental results. ## analysis given the performance gains of our user behavior pre-training methods, we systematically analyze the following scientific questions in this section. q1. how does user modeling contribute to the performances? q2. how do user behavior masking and user behavior generation contribute to the performances? q3. how do total mask ratio α and behavior mask ratio β influence the performances? q4. is the initialization of the autoregression decoder necessary for the pre-training? q5. how does performance change when choosing a siamese or fully separated encoder in fine-tuning? q6. will different pooling methods affect the performances? detailed analyses (a) are conducted below one by one. the seed for all experiments in this section is fixed to 42 for reproducibility. ## related works in this section, we introduce traditional neural news recommendation systems and plm-based neural news recommendation systems. ## conclusion this paper proposes to pre-train with user behavior modeling for better news recommendations. a simple yet efficient user modeling encoder is used for better alignment of user behaviors. on top of this, we further incorporate the user behavior masking task and the user behavior generation task for effective unsupervised pre-training. experiments on real-world news benchmarks show that our works significantly improve the performances of plmbased news recommendations.