Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation (2024)

Abstract: Synthetic users are cost-effective proxies for real users in the evaluation of conversational recommender systems. Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users. We introduce a new protocol to measure the degree to which language models can accurately emulate human behavior in conversational recommendation. This protocol is comprised of five tasks, each designed to evaluate a key property that a synthetic user should exhibit: choosing which items to talk about, expressing binary preferences, expressing open-ended preferences, requesting recommendations, and giving feedback. Through evaluation of baseline simulators, we demonstrate these tasks effectively reveal deviations of language models from human behavior, and offer insights on how to reduce the deviations with model selection and prompting strategies.

## introduction in everyday life, recommendations are often sought through conversations: we ask others for advice on which movies to watch, appliances to buy, or restaurants to explore. such experience is what conversational recommendation systems (crss) seek to provide, by developing autonomous agents that could chat with users, understand their needs, and provide well-tailored recommendations. a core challenge that hinders the advancement of the field is evaluation @xcite . while an ideal approach would involve comprehensive testing with real user interactions, the associated costs and risks drive studies towards proxy methods, which are limited in representing real user evaluation. offline evaluation restricts evaluation to non-interactive modes, allowing only single-turn assessments @xcite @xcite . to enable interactive evaluation, studies have introduced synthetic users. however, they are overly simplified representations of human users, being restricted to binary responses (e.g., yes or no) @xcite or holding 'target' items as if users and agents are playing guessing games @xcite @xcite . other line of work restrict interactions to predetermined rules and templates @xcite . essentially, these user simulators suffer from an inherent constraint: they are static (i.e., confined to a finite set of actions), not generative. recently, large language models (llms) have demonstrated impressive proficiency in conversational tasks @xcite , motivating a growing number of works to explore their capacity to simulate human behavior @xcite @xcite @xcite . agents simulated by llms are generative; conditioned upon profiles and memories, these agents exhibit emergent behaviors that appear believable @xcite @xcite . studies have also explored the use of llms as user simulators for recommender systems @xcite ). an important question in each of these studies is to evaluate how closely these simulators represent humans in the task. while there are automatic evaluation protocols for replicating general human behavior @xcite , no protocol exists in the context of recommendation. in conversational recommendation, the requirements of user simulators are distinct from generalpurpose human simulators. the goal is to simulate a population of users, each with distinct preferences, in a way that these preferences collectively reflect the characteristics of human preferences. real user preferences are highly granular and diverse, shaped by each individual's particular set of traits, interaction history, and circumstances. such uniqueness is reflected in conversational utterances among individuals, each mentioning distinct items, expressing various preferences, and making highly personalized recommendation requests. there are also population-level patterns in preferences, such as users preferring some items over others. protocols in other domains are unsuitable for evaluating the requirements in conversational recommendation, since they do not consider the behaviors driven by personal preferences @xcite . although some work considers the uniqueness of simulated individuals, evaluations are limited to manual case studies @xcite . we propose a new evaluation protocol for measuring the extent to which llm-based simulators can represent users in conversational recommendation. this objective poses new challenges: first, we lack data that maps inputs to ground-truth outputs, such as demographics to survey outcomes @xcite . second, our outcomes are freeform text, unlike previous work where behavioral outcomes are are choices or numerical values for ground truth comparison @xcite . third, the infinite possibilities of conversational trajectories make the concept of 'ground truth' increasingly ambiguous as conversations unfold. we tackle these challenges by decomposing evaluation into five independent tasks, each measuring a key property that a user simulator should exhibit. each task prompts a simulator and stores the outcomes of a population of simulators. the outcomes can then be compared to the human data we curate from four different platforms. the tasks by them-selves do not guarantee simulators to be perfect representations of human users, but rather, capture distortions in simulators, which is the systematic difference from humans @xcite . we demonstrate the effectiveness of these tasks by applying them to baseline simulators and revealing the distortions present in these simulators. we observe that simulators tend to favor mentioning popular items, correlate little with human preferences, exhibit lack of personalization in requests, and occasionally give incoherent feedback. we also identify methods to reduce the gap between simulators and humans, indicating that our evaluation protocol can guide future research in developing more realistic user simulators. our work is summarized as follows: ## evaluation tasks here we first introduce our tasks and later explain the execution of these tasks in section 3. open-ended utterances allow users to express detailed preferences, such as appreciating the cast of a movie while finding the plot uneventful (xia task baselines datasets example prompt (t1) itemstalk di, ih redial, reddit, imdb a person mentions concussion (2015) and jerry maguire (1996) in a conversation about movies and proceeds to mention 2 more. what would these 2 movies be? (t2) binpref di, di + pp movielens pretend to be ms. guzman. you watched the movie whiplash (2014). did you like the movie? answer yes or no. don't say anything else. (t3) openpref di, di + pp imdb pretend to be mr. li. you watched the movie the bellboy (1960). what are your thoughts on this movie? answer should not exceed 809 characters. (t4) recrequest vanilla llm reddit generate a movie recommendation request. include the following movies in your text: oldboy (2003), memento (2000). length of the request is approximately 374 characters. (t5) feedback vanilla llm reddit in the following conversation ... if the recommendation is coherent to your request, answer accept. if the recommendation is incoherent to your request, answer reject. given the vastness of tastes and circumstances, a wide variety of requests may emerge @xcite . we investigate whether requests generated by simulators are as diverse as those of real users. ## methods our protocol treats the design choices of simulators as a black box. we only require that simulators should accept free-form natural language as input and generate language as output. as noted in the introduction, we consider the population of users, since recommender systems are tested against a large group of users. importantly, our focus is not to replicate a fixed pool of users, but to generate a new group of users whose behavior characteristics resemble those of human users. tasks should be zero-shot; simulators should not be trained or conditioned on our tasks, nor be informed about our evaluation metrics. this is to avoid simulators fitting to the tasks instead of performing well in generic situations. ## experiments we summarize our experiment results as follows. finding 1: simulators mention less diverse items compared to real users. our first task, itemstalk, reveals that the distribution of items mentioned by simulators is heavily skewed toward popular items, in contrast to a more even distribution of items mentioned by humans (figure 2 ). we observe the trend across all baselines and datasets, quantified by entropy ( finding 2: prompting with interaction history enhances item diversity. comfortingly, prompting with interaction history yields much higher diversity than prompting with demographic information, and we even observe cases (gpt-4 and text-davinci-003) prompted with interaction history from reddit and imdb slightly exceed the diversity of humans in redial. this suggests that interaction history ('trigger' items) is a strong condition for generating diverse items. finding 3: simulators may poorly represent real user preferences. our next task, binpref, captures simulators failing to represent human preferences. in figure 3 , we sort items in decreasing average rating and plot the positive rate (proportion of simulators that answered 'yes' to whether they liked the movie). positive rates remain constant regardless of human preferences, except gpt-4 + di + pp, where the positive rate decreases as average rating decreases. unexpectedly, higher item frequency (how well known is an item, measured by number of ratings) does not necessarily lead to better preference alignment ( finding 4: adding pickiness personality improves preference alignment. endowing simulators with varying levels of pickiness not only diversifies preferences but also improves correlation, sometimes yielding strong correlation ( ## conclusion we introduce a new protocol for evaluating llms as user simulators for conversational recommendation. we design five evaluation tasks, where each task addresses an essential property for simulators to be realistic user proxies. by running the tasks on simulators, we show how the tasks effectively reveal discrepancies of simulators from real users. our work aims to set a benchmark for evaluating simulators automatically, with future plans to enhance their realism.