Collecting High-quality Multi-modal Conversational Search Data for E -Commerce (2024)

Abstract: Continued improvement of conversational assistants in knowledge-rich domains like E-Commerce requires large volumes of realistic high-quality conversation data to power increasingly sophisticated large language model chatbots, dialogue managers, response rankers, and recommenders. The problem is exacerbated for multi-modal interactions in realistic conversational product search and recommendation. Here, an artificial sales agent must interact intelligently with a customer using both textual and visual information and incorporate results from external search systems, such as a product catalog. Yet, it remains an open question how to best crowd-source large-scale, naturalistic multi-modal dialogue and action data, required to train such an artificial agent. We describe our crowd-sourced task where one worker (the Buyer) plays the role of the customer, and another (the Seller) plays the role of the sales agent. We identify subtle interactions between one worker’s environment and their partner’s behavior mediated by workers’ word choice. We find that limiting information presented to the Buyer, both in their backstory and by the Seller, improves conversation quality. We also show how conversations are improved through minimal automated Seller “coaching”. While typed and spoken messages are slightly different, the differences are not as large as frequently assumed. We plan to release our platform code and the resulting dialogues to advance research on conversational search agents.

## introduction in recent years, researchers have investigated new approaches to build automated agents capable of naturalistic conversations satisfying complex information needs. the need for such assistance is particularly acute in domains like e-commerce, where customers may even know what questions to ask when shopping. creating an automated agent to help such customers is challenging, as it must serve as a natural conversational interface to many specialized and general data sources, while maintaining context to return valuable responses or make proactive suggestions. in high-stakes domains like e-commerce, experimentation on real customers is risky, presenting a significant barrier to training and validating such conversational agents. most taskoriented conversation systems, especially for e-commerce, have been evaluated on synthetic data even though such data may not reflect real world conversations @xcite . thus there is a clear need for high-quality and realistic data to be collected to allow training more natural automated agents. moreover, it isn't clear how to generate natural data that reflects real-world conversations in domains like e-commerce. although large language models such as chatgpt can be used to generate conversation, these are not realistic as shopping conversations are dynamic and customer goals can change as new information is presented @xcite . we explore crowd task designs needed to collect the right data to train conversational search and recommender agents, roughly following a theoretical framework @xcite . such agents combine aspects of a mixed-initiative chatlike dialogues with goal oriented systems, creating a number of unique challenges. consider example 1. the customer seeks help from the agent with a specific goal, buying a tv set, but the conversation is open-ended. unlike, say, airline reservation systems, there are few constraints on the utterances. in addition, agent and customer share initiative. the agent must choose from several information sources to help construct a reply. figure 1 shows a standard framework to provide such an experience. a language understanding module extracts information from customer utterances, used by the dialog manager to query sources, including product catalogs, qa databases, etc. a ranker selects among candidates to produce each system response. example 1: an example dialog illustrating a desirable e-commerce conversation with an automated agent (when we refer to an agent, we mean an automated agent). to train such an agent, we need a corpus of highquality naturalistic, free-form, but goal-oriented conversations labeled with each action's intent and overall conversation success. recently one such dataset was released @xcite with a large number of conversations related to furniture and fashion. however, there is no exploration of the conversations' naturalness, or of how the multimodal environment affects the results. moreover, it isn't clear that such a corpus is ecologically valid @xcite , i.e., it isn't clear that the information and tools provided to the workers realistically emulate how customers would interact with automated or human sales agents. to ensure that, we aim here to understand how the conversation participants' multi-modal environment affects the conversation, and to add to the available conversational search and recommendation data. we review prior work in the next section to put our contributions in context. section 3 describes how we crowd-sourced conversations. section 4 covers the development of our crowd-sourced annotations and automated measures of the conversation. we analyze the collected conversation data in section 5, and demonstrate the effectiveness of the collected data to enable a robust conversational search and recommendation agent. finally, we discuss future work and potential extensions in section 6. contributions and research questions multimodal, conversational search opens up many questions. we focus on how the workers' environmentthe information and tools provided to each worker -affects their interactions with each other. to that end, we aim to answer four research questions: r1 how does one worker's environment affect their behavior and language use? r2 can one worker's environment alter the other's behavior including language and feature use? r3 how much do spoken and typed utterances differ in a conversational search environment? r4 what buyer and seller environments yield the most realistic conversations? our work for the first time explores and analyzes the most effective conditions for priming workers in large-scale, crowd-sourced conversational data collection, with multiple interaction modalities, and introduces quantitative evaluation metrics for dialogue quality, which, as we demonstrate, could be used to train a conversational e-commerce search and recommendation agent. ## related work automated conversational agents have been an active area of study, and their use has exploded following the success of voice-based conversational agents such as siri, alexa, and google home assistant. a long-term goal for dialogue systems is to coherently and engagingly converse with humans on a variety of topics @xcite @xcite . however, such systems require extensive engineering or extensive training data collection and annotation, or both. below, we review prior approaches to collecting and annotating data for training conversational agents. we focus mostly on task-oriented agents in complex domains such as search and recommendation, i.e., information-oriented and transactional tasks @xcite . ## evaluation noting that evaluating conversations remains difficult and subjective, we use a variety of measures to understand workers' behavior in our crowd-sourced conversations. we quantify conversations with automated linguistic measures, language modeling, worker surveys, and manual annotation. ## results below, we answer our research questions. we show in particular that buyers' priming (i.e., their environment) significantly alters their behavior and language use (r1), that buyers' environment significantly alters sellers' behavior (r2). finally, we find that while there are some differences between spoken and typed conversational messages, these differences are not as large as would be expected from studies of keyword-style spoken and typed search queries (r3). we address what are the "best" conditions (r4) throughout this section. ## conclusions we have highlighted several important results that should advance future efforts to crowd-source conversations for effective conversational multi-modal search. first, spoken and typed messages are not as different as previously thought. we attribute this to the conversational nature of the task. this suggests that transfer learning approaches that take advantage of the more plentiful text-based conversations are a promising avenue for voice systems as well. importantly, conversational partner's behaviorboth "private" behaviors like mouse-overs as well as the language used to communicate-affects the other partner's behavior as well, and we found we can influence both behaviors through priming. our results show that task design must both direct the desired behaviors as much as possible (e.g., seller coaching) but must avoid providing too much information. we should look for opportunities to influence seller behavior in the buyer's environment as well. we emphasize that the more structure the seller has, the better the resulting conversations. our findings have implications for voice assistants as well: workers will do what we've taught them to do, and ask questions only about the information we present to them. therefore, to enable good conversational systems for search and exploration, strategies to prime customers with a knowledge of the actions they can take, and the information they can obtain are critical. for instance, if a system presents "price" as a key attribute to customers, our results show that customers are more likely to focus on price in their product exploration.