Salespeople vs S ales B ot: Exploring the Role of Educational Value in Conversational Recommender Systems (2023)

Abstract: Making big purchases requires consumers to research or consult a salesperson to gain domain expertise. However, existing conversational recommender systems (CRS) often overlook users’ lack of background knowledge, focusing solely on gathering preferences. In this work, we define a new problem space for conversational agents that aim to provide both product recommendations and educational value through mixed-type mixed-initiative dialog. We introduce SalesOps, a framework that facilitates the simulation and evaluation of such systems by leveraging recent advancements in large language models (LLMs). We build SalesBot and ShopperBot, a pair of LLM-powered agents that can simulate either side of the framework. A comprehensive human study compares SalesBot against professional salespeople, revealing that although SalesBot approaches professional performance in terms of fluency and informativeness, it lags behind in recommendation quality. We emphasize the distinct limitations both face in providing truthful information, highlighting the challenges of ensuring faithfulness in the CRS context. We release our code and make all data available.

## introduction conversational recommender systems (crs) use multi-turn dialog to understand user preferences, gather information, and provide suitable recommendations @xcite . they have gained significant attention from academia and industry due to their flexibility compared to one-shot recommender systems. crs allow users to dynamically refine their preferences and express feedback on recommended items through chat. while many traditional crs are built under a "system ask-user answer" paradigm @xcite , some recent works have identified the critical need to support mixed-type, mixed-initiative dialog and assist users with underspecified goals @xcite @xcite . however, most crs remain focused on domains involving content recommendation such as movies, books, music, news, etc, which require a different type of recommendation strategy than the ecommerce space where consumers often need a certain level of background knowledge to understand their options @xcite . in content recommendation domains, crs can achieve success by questioning a user about previous content consumption and retrieving similar content. this strategy is not valid for the sale of complex products, as prior user habits do not inform a buyer's item-specific preferences. in this work, we focus on complex products with multiple attributes that would usually require significant expertise or a salesperson consultation to make an informed purchase decision on -e.g., tvs, guitars, etc. we associate the ambiguous user goals often present in this real-life setting with the lack of knowledge about the product domain and suggest the need for the agent to provide educational value in addition to fulfilling the recommendation objective. we address this new problem space by introducing salesops, a framework that facilitates the recreation of realistic crs scenarios and agent evaluation, as shown in figure 1 . our method incorporates several novel features. we leverage recent progress made in large language models (llms), which has enabled increasingly powerful conversational agents, to build a pair of agents -salesbot and shopperbot which can simulate either side in the salesops framework. thus, facilitating evaluation in any of the following settings: human-human, human-bot, bot-human, and bot-bot. we critically examine the components of these agents to understand where modern llms have the most impact on crs. we recruit 15 professional salespeople to study the gap between human experts and salesbot at complex product conversational recommendations. the results reveal that although salesbot matches professionals in tone politeness and educational quality, there remain minor gaps in the quality of recommended products. we also perform a preliminary analysis of faithfulness within the salesops framework, which is important in domains involving generative ai as they enter the applied setting. our nli-based analysis @xcite reveals that salespeople can use strategies that may seem unfaithful, to upsell or simplify technical details. these results highlight the practical challenges of implementing robust faithfulness checks in domains like conversational sales. ## salesops framework we now describe the salesops framework, allowing us to study crs systems in terms of educational and recommendation objectives for complex product scenarios. in salesops, two actors -the seller and the shopper -have a conversation that begins with a shopper request and ends once the seller makes a product recommendation that the shopper accepts. each actor gets access to specific content elements that assist them in completing the task. we first describe the three content elements and the procedure used to generate them at scale, and then we describe the constraint we put on actors' access to the content to create realistic sales conversations. ## bots implementation we now present shopperbot and salesbot, the llm-based implementations (chatgpt in our experiment) that can simulate both sides of the sale-sops framework. ## evaluation criteria along with the salesops framework, we propose a multi-dimensional evaluation that defines success for the seller along three axes: (1) recommendation quality, which verifies whether the recommendations of the seller are compatible with the shopper's preferences, (2) informativeness, which checks whether the seller provides educational value to the shopper, and (3) fluency which evaluates the seller's ability to communicate professionally and concisely. we next define the metrics -both automatic and manual -used to evaluate each axis. recent work shows the promise of using llms for evaluation across many nlp tasks @xcite @xcite . following this thread of work, we leverage gpt-4 (openai, 2023) for automatic evaluation purposes in several of our proposed metrics. ## ablation study salesbot contains 4 llm-based components, which we swap with baselines to understand the impact of using an llm for each component. ## salespeople vs salesbot we aim to study the qualitative differences between salesbot and professional salespeople to comprehend the effect of deploying such systems in realworld settings @xcite , and perform an extensive human evaluation of salesbot and salespeople within the salesops framework. ## conclusion in this paper we introduce salesops, a flexible framework for simulating realistic crs scenarios in the context of complex product sales, and propose an evaluation paradigm for such systems. our framework provides researchers and practitioners with a valuable tool for assessing the effectiveness and performance of conversational agents in sales settings. by developing salesbot and shopperbot within this framework, we gain insights into the individual components and their impact on conversational performance. through a comprehensive human study, we identify gaps in product recommendation quality and provide a faithfulness analysis of both automated and human sellers. these contributions advance the understanding and development of crs, paving the way for improved sales interactions and user experiences.