Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset (2024)

Abstract: Conversational recommender systems are an emerging area that has garnered increasing interest in the community, especially with the advancements in large language models (LLMs) that enable sophisticated handling of conversational input. Despite the progress, the field still has many aspects left to explore. The currently available public datasets for conversational recommendation lack specific user preferences and explanations for recommendations, hindering high-quality recommendations. To address such challenges, we present a novel conversational recommendation dataset named PEARL, synthesized with persona- and knowledge-augmented LLM simulators. We obtain detailed persona and knowledge from real-world reviews and construct a large-scale dataset with over 57k dialogues. Our experimental results demonstrate that PEARL contains more specific user preferences, show expertise in the target domain, and provides recommendations more relevant to the dialogue context than those in prior datasets. Furthermore, we demonstrate the utility of PEARL by showing that our downstream models outperform baselines in both human and automatic evaluations. We release our dataset and code.

## introduction recently, conversational recommender system (crs) has become an emerging research topic, which aims to elicit user preferences and offer personalized recommendations by engaging in interactive conversations. towards this goal, an increasing emphasis has been placed on constructing highquality dataset @xcite @xcite . existing conversational recommendation datasets are mainly collected via crowdsourcing, which is to gather interactions between two crowdworkers where one plays a role as a user (i.e., recommendation seeker) and the other pretends as a recommender. however, existing dialogues have several limitations that impede downstream crs models from delivering satisfactory user experiences. first, user preferences expressed in the existing crs datasets are often less specific. an example includes dialogues with statements like, "i like most genres" as shown in figure 1 . such dialogues lead downstream models to offer recommendations that are generic and less personalized @xcite . this happens as crowdworkers, playing the role of users, often lack specific preferences during their tasks, unlike most real-world users who seek personalized recommendations. second, conversational recommendation dialogues often provide suboptimal recommendations and insufficient explanations alongside recommendations, due to the limited knowledge of crowdworkers @xcite . a specific example is dialogues containing utterances like "let me see... how about tropic thunder?" as shown in figure 1 . such responses can lead to crs models generating less accurate and relevant suggestions. additionally, the absence of explanations alongside recommendations can be a crucial hurdle, preventing users from grasping the items and the rationale behind the recommendations. in this paper, we present a conversational recommendation dataset named pearl (persona and knowledge augmented recommendation dia-logues) that addresses the limitations of existing datasets. we transform reviews into persona and item knowledge and incorporate large language model (llm)-based simulators augmented with persona and knowledge. each simulator is designed to enhance the preference specificity and informativeness of the collected data, respectively. the user simulator plays the role of a recommendation seeker and is equipped with a persona, which is a set of sentences describing features that the user likes and dislikes. each persona can help the simulator express a distinct user preference of a single real-world user as it is constructed based on the item reviews written by the same user. by simulating users with distinct preferences and specific needs, we can generate dialogues with consistent and clear preferences. the recommender simulator is designed to emulate a recommender with domain knowledge. this addresses the lack of proper recommendations and explanations in existing datasets. specifically, we incorporate item reviews which can not only provide basic information about items but also reveal soft attributes of items that can be only described through experience (e.g., "feel-good movies" or "upbeat music") which could be crucial information in the users' decision-making process. our collected dataset includes over 57.2k dialogues simulating over 4k users and covering more than 9k items. our human evaluation results show that our synthesized dataset is preferred by human raters compared to other crowdsourced dialogues. moreover, our dataset covers a broader spectrum of user needs as we utilize a large set of authentic reviews written by diverse users. we also conduct extensive experiments assessing the utility of pearl through human evaluation and automatic evaluation. the results demonstrate that crs models trained on pearl show competitive or better performances in recommendation and response generation tasks compared to models trained on human-annotated datasets. additionally, human judges consistently favor responses from models trained on pearl over those from models trained on crowdsourced datasets, across all metrics. furthermore, our experimental results empirically validate that crs models trained on pearl generalize better to unseen dialogues than the models trained on existing datasets. ## pearl construction we construct pearl, a large-scale recommendation dataset covering diverse preferences and detailed item explanations through five steps: (1) grouping real-world reviews with two axes, which are user and item, ( §3.1), (2) equipping a user simulator with preferences of a real-world user using reviews ( §3.2), (3) infusing item knowledge extracted from reviews to a recommender simulator ( §3.3), (4) inferring the simulators to derive a dialogue ( §3.4), and (5) filtering dialogues ( §3.5). while our dataset construction process is domainindependent, we validate it in the movie domain due to its extensive prior attention @xcite @xcite , making it easier to evaluate. the overview of pearl construction process is shown in figure 2 and an example of pearl is in ## conclusion in this work, we introduce pearl, a largescale conversational recommendation dataset constructed using llm simulators augmented with persona or knowledge from real-world reviews. our comprehensive experiments validate pearl's superior quality and utility in training models compared to existing datasets. pearl paves the way for future research in developing effective conversational recommender systems, particularly those emphasizing explainability, knowledge retrieval, and reasoning abilities. also, pearl offers a practical opportunity for developing small and deployable systems capable of handling specific user feedback and providing satisfactory recommendations. dialogue context ... seeker: i like both, but i'm looking for something super hero related, like guardians of the galaxy. recommender: oh! i have a few ideas of what you have in mind. are comic book characters interesting at all? seeker: yes! i love comic book characters, like spider man, ironman and captain america. recommender: okay. would you like to watch this trailer for the avengers? seeker: oh that would be great! model responses gpt-3.5: i would recommend you watch the trailer for "avengers: endgame." it features all your favorite comic book characters like iron man, captain america, and spider-man, and it's a super hero-related movie just like guardians of the galaxy. enjoy watching the trailer! bart-redial: okay. i would also recommend 'the avengers (2012)' bart-pearl (ours): how about 'the avengers ( 2012 )'? it's an action-packed superhero movie with a great cast and some really interesting characters. ground-truth: i haven't actually watched it yet. it is the one with that thanos character and the justice league, right? the full example is in