User Simulator Assisted Open-ended Conversational Recommendation System (2023)

Abstract: Conversational recommendation systems (CRS) have gained popularity in e-commerce as they can recommend items during user interactions. However, current open-ended CRS have limited recommendation performance due to their short-sighted training process, which only predicts one utterance at a time without considering its future impact. To address this, we propose a User Simulator (US) that communicates with the CRS using natural language based on given user preferences, enabling long-term reinforcement learning. We also introduce a framework that uses reinforcement learning (RL) with two novel rewards, i.e., recommendation and conversation rewards, to train the CRS. This approach considers the long-term goals and improves both the conversation and recommendation performance of the CRS. Our experiments show that our proposed framework improves the recall of recommendations by almost 100%. Moreover, human evaluation demonstrates the superiority of our framework in enhancing the informativeness of generated utterances.

## related work conversational recommendation system current crs studies can be roughly categorized into two directions @xcite : (1)attributecentric crs @xcite @xcite . these systems ask questions about the user preferences of certain attributes or make recommendations at each turn and gradually narrow down the hypothesis space of items to make optimal recommendations. these studies focus on the recommendation part and use question/answer templates with attribute or item slots. they often use reinforcement learning to achieve better recommending and asking policies. user simulator traditional uss are rulebased such as the agenda-based user simulator (abus) @xcite . for different tasks, abus needs to design different hand-crafted structures, which poses challenges in scenario shifting. data-driven us @xcite is another line of work. a seq2seq model is used to generate semantic-level dialog acts @xcite @xcite or natural languages @xcite . however, most of the uss are de-signed for task-oriented dialog systems and cannot be directly used for crs. to the best of our knowledge, our work is the first to explore us for openended crs that can generate consistent responses based on certain user preferences. ## conclusion in this paper, we propose a framework to be packed with any crs to improve its recommendation accuracy and language informativeness. we first build a user simulator for open-ended crs with three preference-aware modules to give the appropriate feedback to the crs based on certain user preferences. we then fine-tune a pre-trained crs with reinforcement learning based on its interaction with the us with two types of designed rewards. experiments demonstrate that our framework can significantly improve the recall of the recommendation, and human evaluation shows that the generated language is more informative with more descriptions of the recommended items. for future work, the first is to use joint optimization of crs and us to further improve the interactive qualities, and the second is to explore the generalizability of the framework to other domains of recommendation.