Enhancing Large Language Model Based Sequential Recommender Systems with Pseudo Labels Reconstruction (2024)

Abstract: Large language models (LLMs) are utilized in various studies, and they also demonstrate a potential to function independently as a recommendation model. Nevertheless, training sequences and text labels modifies LLMs’ pre-trained weights, diminishing their inherent strength in constructing and comprehending natural language sentences. In this study, we propose a reconstruction-based LLM recommendation model (ReLRec) that harnesses the feature extraction capability of LLMs, while preserving LLMs’ sentence generation abilities. We reconstruct the user and item pseudo-labels generated from user reviews, while training on sequential data, aiming to exploit the key features of both users and items. Experimental results demonstrate the efficacy of label reconstruction in sequential recommendation tasks.

## introduction recommender systems have achieved significant advancement, becoming essential in various domains such as e-commerce, streaming services, and social media. despite their widespread application, traditional methods face several limitations, particularly in extracting and effectively utilizing textual information, such as user reviews, in addition to interaction data. traditional models, @xcite @xcite @xcite , while effective in leveraging numerical data such as ratings and purchase history, often struggle with capturing the nuanced contextual information present in user reviews and other textual data. this limits their ability to fully understand user preferences and provide highly personalized recommendations. large language models (llms), like gpt @xcite , have transformed the field of recommender systems by excelling in understanding and generating natural languages. they effectively address the limitations of traditional models in handling textual data and are actively studied for their potential in various recommendation tasks. several efforts have leveraged llms for zero/few-shot recommendation by incorporating user history and candidate items as input prompt @xcite @xcite . additionally, llms have been employed to address cold-start problem @xcite and have been utilized for data augmentation purposes @xcite @xcite . moreover, studies that have aimed to train llms via efficient measures @xcite @xcite have demonstrated their potential to function as recommendation models. llm-based recommender systems leverage pretrained knowledge to understand diverse textual inputs and generate rich textual representations @xcite . this allows llms to provide contextually relevant suggestions that align closely with user preferences. however, training llms on recommendation datasets can disrupt their pre-trained weights, especially their generation capabilities @xcite . therefore, new methods are needed to use llms effectively while preserving their strengths. our proposed methodology addresses such challenges by leveraging llms' advanced textual representation generation in a sequential recommendation framework. to preserve the pre-trained knowledge of the llm, we implemented separate embeddings for users and items while freezing the llm's transformer layers and the word embedding. this ensures that the model retains its ability to construct coherent sentences. to train user and item embedding, we generated pseudo-labels for users and items based on their reviews. finally, we trained the model using both sequential data and the pseudo-labels simultaneously. the main contributions of this paper are: (1) proposing a reconstruction-based llm model (relrec) that captures and effectively integrates user/item label features for next item predictions; ## proposed approach in this study, we propose relrec, a model composed of two primary components: the creation of user/item embeddings with pseudo-labels, and the training process that incorporates both sequential data and textual labels. ## conclusion in this paper, we proposed relrec, a reconstruction based llm recommendation model which integrates features of user/item text labels to the next item prediction. our proposed model, relrec, effectively utilizes pseudo-labels generated from user reviews to capture nuanced information and enhance recommendation accuracy by label reconstruction. experimental results demonstrates the effectiveness of relrec.