Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning (2023)

Abstract: Recent studies have proposed unified user modeling frameworks that leverage user behavior data from various applications. Many of them benefit from utilizing users’ behavior sequences as plain texts, representing rich information in any domain or system without losing generality. Hence, a question arises: Can language modeling for user history corpus help improve recommender systems? While its versatile usability has been widely investigated in many domains, its applications to recommender systems still remain underexplored. We show that language modeling applied directly to task-specific user histories achieves excellent results on diverse recommendation tasks. Also, leveraging additional task-agnostic user histories delivers significant performance benefits. We further demonstrate that our approach can provide promising transfer learning capabilities for a broad spectrum of real-world recommender systems, even on unseen domains and services.

## introduction recent advances in user modeling have focused on constructing unified user models to be directly adapted to diverse applications. many of them leverage natural language or plain text data, which enables general-purpose applicability among various domains and systems @xcite @xcite @xcite . these strategies pave a much more efficient way for service owners to quickly adapt to various task scenarios by tuning one single model, bringing performance improvement across whole systems in parallel. based on the recent explosions of sequence prediction models in many domains @xcite @xcite @xcite , it is natural to ask § both authors contributed equally to this research. correspondence to: <ky.shin@navercorp.com>. some recent studies confirm that continued pretraining of language model on few task-specific data drawn from the target task distribution, or data similar to a target domain can provide significant benefits to solve downstream classification tasks @xcite @xcite . interestingly, @xcite go further and validate that language models trained from scratch on task-specific or task-agnostic data foot_0 -data from other downstream tasks-can rival standard webtext language models. another line of research provides mathematical explanations of how language model pretraining can improve performances on downstream tasks @xcite . more specifically, @xcite reformulate classification tasks as sentence completion tasks, thus demonstrating that linear classification using output features from fixed gpt-2 @xcite , i.e., no finetuning, also guarantees to solve sentence classification tasks. motivated by these works, we introduce a new method called lmrec, which jointly trains language model and recommendation task objectives from user behavior histories transformed as plain text format. as illustrated in figure 1 , our approach is conceptually simple but practically effective. we first investigate if the recommender system jointly trained with the language modeling objective on task-specific data can enrich the user/item representations, thus providing better generalization even for unseen downstream tasks ( our major findings are as follows: jointly training language modeling and recommendation task objectives improve recommender systems. language modeling on the user history can produce rich user/item representations for diverse applications. these results are consistent with the effect of task-adaptive pretraining in the previous research @xcite @xcite . furthermore, our approach also boosts the transfer learning capability of the recommendation model. extensive experimental results show the efficacy of our approach compared to training without language model objectives ( ## related work any model that trains a text-based user model to adapt to unseen domains/systems can be viewed as prior work of our research. this line of work has been recently explored since learning text representation has been rapidly developed in the decade. in this context, @xcite and @xcite are the earliest work we are aware of. they train the model through critical word matching in user logs and then finetune models to the downstream tasks. first, the word (item) embeddings are precomputed using pretrained language models (plms). the sequence of item embeddings is then passed to the encoder to produce user representations. recently, some researchers propose to use behavior history as plain text data @xcite @xcite . @xcite and @xcite introduce a contrastive learning framework on multiple service domains, and perform transfer learning across various downstream tasks. another line of work @xcite tries to construct personalized prompts for building versatile framework, i.e., "here is the history of {gender} {age}: {history from all services}, the user is now recommended a {item}". this approach profits from the methods that utilize language models such as gpt-2 (radford et al., 2019), t5 @xcite , and m6 @xcite . their plm-based approach can be generalized to various applications, with the ability to perform zero-shot learning. @xcite is the only work that trained the whole encoder from scratch rather than using plms. we refer readers to @xcite and @xcite for an overview of this line of work. a related idea to our work is the training language model on task-specific or task-agnostic corpora. it has been shown to be beneficial in a variety of works @xcite @xcite @xcite . @xcite continue pretraining of lm on task-specific data and show it can improve the downstream performances of standard webtext language models. @xcite point out that the effect of pretraining on standard webtext data may have been overestimated. they show that models trained only on task-specific data comparably perform to existing webtext language models. on the one hand, a line of research jointly trains language models on taskspecific data during finetuning to avoid catastrophic forgetting @xcite . some of the works above also investigate if the models pretrained on task-agnostic data can be effective for downstream tasks. @xcite and @xcite show domain-adaptive pretraining further improves the performance of pretrained language models. recently, @xcite have observed that pretraining on task-agnostic data can provide a significant advantage compared to standard webtext data. these findings give huge insight into our research. note that our work aims at extending the potential of language modeling that has been successfully used for diverse applications to recommender systems. ## conclusion recent works have built text-based user models and demonstrated that the rich nature of text information in any domain or system could be a valuable foundation for user modeling. our primary contribution is jointly optimizing the language modeling and recommendation task objectives and successfully tackling a broad spectrum of diverse recommendation tasks, including transfer learning for unseen domains and systems. overall, our analysis sheds remarkable insights on user representation learning through user behavioral corpora.