Stop Playing the Guessing Game! Evaluating Conversational Recommender Systems via Target-free User Simulation (2025)

Abstract: Recent developments in Conversational Recommender Systems (CRSs) have focused on simulating real-world interactions between users and CRSs to create more realistic evaluation environments. Despite considerable advancements, reliably assessing the capability of CRSs in eliciting user preferences remains a significant challenge. We observe that user-CRS interactions in existing evaluation protocols resemble a guessing game, as they construct target-biased simulators pre-encoded with target item knowledge, thereby allowing the CRS to shortcut the elicitation process. Moreover, we reveal that current evaluation metrics, which predominantly emphasize single-turn recall of target items, suffer from target ambiguity in multi-turn settings and overlook the intermediate process of preference elicitation. To address these issues, we introduce PEPPER, a novel CRS evaluation protocol with target-free user simulators that enable users to gradually discover their preferences through enriched interactions, along with detailed measures for comprehensively assessing the preference elicitation capabilities of CRSs. Through extensive experiments, we validate PEPPER as a reliable simulation environment and offer a thorough analysis of how effectively current CRSs perform in preference elicitation and recommendation.

## introduction conversational recommender systems (crss) have played an increasingly important role in enhancing personalized experiences by providing tailored recommendations through interactive dialogues @xcite @xcite . throughout the interaction, these systems are expected to perform two key tasks: (1) preference elicitation -exploring and uncovering user preferences by encouraging them to express their likes and dislikes, and (2) recommendation -retrieving personalized items based on the preferences inferred from the dialogue. in the field of crss, automatically evaluating the system's capability has remained challenging @xcite @xcite @xcite . conventional offline approaches relying on static, pre-collected dialogues from datasets often neglect the system's responsibility to dynamically shape the dialogue itself, whereas evaluating with real user interactions is costly and timeconsuming @xcite @xcite . ## pepper: target-free crs evaluation guided by the limitations of existing evaluation protocols, we introduce pepper, a novel evaluation protocol designed to comprehensively assess both preference elicitation and recommendation abilities of crss, addressing key shortcomings of prior approaches. specifically, it incorporates two key components: (1) target-free user simulators with richly expressed preferences derived from real user interaction histories and reviews, and (2) preference elicitation metrics that thoroughly measure a crs's ability to uncover diverse user preferences and deliver accurate recommendations. ## experiments we conduct comprehensive experiments to demonstrate the reliability of pepper. implementation of user simulation and qualitative evaluation is detailed in appendix a.4 and a.5.1, respectively. ## conclusion in this work, we propose pepper, a novel evaluation protocol that comprehensively assesses both preference elicitation and recommendation accuracy in crss. pepper incorporates target-free user simulators, along with both quantitative and qualitative metrics, targeting four distinct aspects of the preference elicitation process. through extensive experiments, we demonstrate the effectiveness of pepper, offering valuable insights into the limitations of existing crs evaluation protocols.