PTUM : Pre-training User Model from Unlabeled User Behaviors via Self-supervision (2020)

Abstract: User modeling is critical for many personalized web services. Many existing methods model users based on their behaviors and the labeled data of target tasks. However, these methods cannot exploit useful information in unlabeled user behavior data, and their performance may be not optimal when labeled data is scarce. Motivated by pre-trained language models which are pre-trained on large-scale unlabeled corpus to empower many downstream tasks, in this paper we propose to pre-train user models from large-scale unlabeled user behaviors data. We propose two self-supervision tasks for user model pre-training. The first one is masked behavior prediction, which can model the relatedness between historical behaviors. The second one is next K behavior prediction, which can model the relatedness between past and future behaviors. The pre-trained user models are finetuned in downstream tasks to learn task-specific user representations. Experimental results on two real-world datasets validate the effectiveness of our proposed user model pre-training method.

## introduction user modeling is a critical technique for many personalized web services such as personalized news and video recommendation @xcite . many existing methods model users from their behaviors @xcite . for example, @xcite proposed a youtubenet model for video recommendation, which models users from their watched videos and search tokens. @xcite proposed a deep interest network (din) for click-through rate (ctr) prediction, which models users from user behaviors on the e-commerce platform based on their relevance to the candidate ads. @xcite proposed to use a gru network for news recommendation, which models users from their clicked news. however, these methods mainly rely on sufficient labeled data to train user models, and their performance may be not optimal when training data is scarce. in addition, they only model task-specific user information and do not exploit the universal user information encoded in user behaviors. in recent years, pre-trained language models such as elmo @xcite , bert @xcite and xlnet @xcite have achieved great success in many nlp tasks, such as reading comprehension and machine translation. many language models are pre-trained on a large unlabeled corpus via self-supervision tasks such as masked lm and next sentence prediction to model the contexts @xcite . these language models can learn universal language representations from large unlabeled corpus and empower many different downstream tasks when the labeled data for these tasks is insufficient @xcite . motivated by pre-trained language models, in this paper we propose pre-trained user models (ptum), which can learn universal user models from unlabeled user behaviors. 1 we propose two self-supervision tasks for user model pre-training. the first one is masked behavior prediction, which aims to infer the randomly masked behavior of a user based on her other behaviors. it can help the user model capture the relatedness between historical user behaviors. the second one is next k behaviors prediction, which aims to predict the k future behaviors based on past ones. it can help the user model capture the relatedness between past and future behaviors. the pre-trained user model is further fine-tuned in downstream tasks to learn task-specific user representations. we conduct experiments on two real-world datasets for user demographic prediction and ads ctr prediction. the behavior 1 behavior 2 behavior n ... user embedding behavior encoder behavior encoder behavior encoder ... ... user encoder behavior embedding position 1 position 2 position n ... results validate that our ptum method can consistently boost the performance of many user models by pre-training them on unlabeled user behaviors. ## conclusion in this paper, we propose an effective user model pretraining method ptum which can pretrain user models from unlabeled user behaviors. in our method, we propose two self-supervision tasks for user model pre-training. the first one is masked behavior prediction and the second one is next k behaviors prediction, which can help user models capture the relatedness among historical behaviors and the relatedness between past and future behaviors. extensive experiments on two real-world datasets for different tasks show that pre-training user models can consistently boost the performance of various user modeling methods.