R ec GPT : Generative Pre-training for Text-based Recommendation (2024)

Abstract: We present the first domain-adapted and fully-trained large language model, RecGPT-7B, and its instruction-following variant, RecGPT-7B-Instruct, for text-based recommendation. Experimental results on rating prediction and sequential recommendation tasks show that our model, RecGPT-7B-Instruct, outperforms previous strong baselines. We are releasing our RecGPT models as well as their pre-training and fine-tuning datasets to facilitate future research and downstream applications in text-based recommendation. Public “huggingface” links to our RecGPT models and datasets are available at: https://github.com/VinAIResearch/RecGPT

## introduction recommendation systems assist in comprehending user preferences and offering suitable content suggestions for users @xcite @xcite . currently, recommendation systems have found wide applications across various domains, such as e-commerce @xcite , news @xcite , and movies @xcite . the evolution of recommendation systems has witnessed a shift from fundamental methods to more sophisticated and modern approaches. conventional methods mine interaction matrices to exploit user-item relationships @xcite @xcite , and subsequently, they incorporate deep learning techniques such as cnn and rnn to extract item features and capture user preferences @xcite . however, this task-specific setting suffers from data sparsity, a lack of flexibility to capture fluctuations in user preferences over time, and challenges in scaling to a large number of users and extensive datasets. later works, inspired by attention mechanisms and the transformers architecture @xcite , model user histories as sequences of items and then encode information in dense vectors @xcite @xcite . with the advancement of large language models (llms), recent works leverage the capacity of llms in understanding user preferences @xcite . the model p5 @xcite , which represents users and items by ids, endeavors to aggregate recommendation tasks under a unified conditional generation model based on t5 @xcite . in addition, @xcite evaluate the potential usage of chatgpt in different recommendation tasks. more recently, @xcite fine-tune llama @xcite with @xcite for sequential recommendation. recommendation tasks frequently exhibit shared characteristics such as user sets, item sets, and interactions, thus suggesting the possibility of training a unified model for multiple tasks, as opposed to employing distinct models for each task. adopting a single model approach, as done in p5, not only encourages model generalization but also fosters collaborative learning across tasks. however, representing users and items by ids, as in p5, may not fully align with the textual understanding capability of llms. it might be more effective to represent items by their textual descriptions and users by their text-based interaction history with items. in this paper, (i) we introduce the first domainadapted and fully-trained llm series named recgpt for text-based recommendation, which comprises the base pre-trained model recgpt-7b and its instruction-following variant, recgpt-7b-instruct. in this context, we pre-train recgpt-7b using a relatively large recommendation-specific corpus of 20.5b tokens, while recgpt-7b-instruct is the model output by further fine-tuning recgpt-7b on a dataset of 100k+ instructional prompts and their responses. (ii) we conduct experiments for rating prediction and sequential recommendation ## our model recgpt this section describes the data and outlines the architecture and optimization setup used for recgpt. ## experiments we conduct experiments to compare our recgpt-7b-instruct with strong baselines for rating prediction and sequential recommendation tasks. ## conclusion we have introduced the first domain-adapted and fully-trained llms for text-based recommendation, which include the base pre-trained recgpt-7b and its instruction-following variant, recgpt-7b-instruct. we demonstrate the usefulness of recgpt by showing that recgpt-7b-instruct outperforms strong baselines in both rating prediction and sequential recommendation tasks. through the public release of recgpt models and the pre-training and supervised fine-tuning datasets, we hope that they can foster future research and applications in text-based recommendation.