PEARL : Preference Extraction with Exemplar Augmentation and Retrieval with LLM Agents (2024)

Abstract: Identifying preferences of customers in their shopping journey is a pivotal aspect in providing product recommendations. The task becomes increasingly challenging when there is a multi-turn conversation between the user and a shopping assistant chatbot. In this paper, we tackle a novel and complex problem of identifying customer preferences in the form of key-value filters on an e-commerce website in a multi-turn conversational setting. Existing systems specialize in extracting customer preferences from standalone customer queries which makes them unsuitable to multi-turn setup. We propose PEARL (Preference Extraction with ICL Augmentation and Retrieval with LLM Agents) that leverages collaborative LLM agents, generates in-context learning exemplars and dynamically retrieves relevant exemplars during inference time to extract customer preferences as a combination of key-value filters. Our experiments on proprietary and public datasets show that PEARL not only improves performance on exact match by ~10% compared to competitive LLM-based baselines but additionally improves inference latency by ~110%.

## introduction large selection, attractive pricing and convenience have made online shopping very popular in recent years. however, traditional e-commerce services still offer search-based interface which is inadequate for broad, ambiguous and upper funnel user queries. absence of a conversational interface often leaves customers feeling the need of humanlike assistance to explain their requirements and navigate towards the right set of products @xcite @xcite @xcite @xcite . with the emergence of generative artificial intelligence (genai) in recent years, much research efforts have been devoted on building multi-turn conversational chat-bots that can serve as virtual shopping assistant, akin to the trained sales agents commonly found in physical stores. in contrast to traditional search-based ecommerce services that operate on single-shot queries, multi-turn conversations require the identification of an evolving set of user preferences embedded within the dialogue. the primary objective of the chatbot is to extract customer preferences from the conversation and map them to preference filters (e.g. brand, cpu, price etc.). extracting preference filters from multi-turn conversation is challenging for several reasons. firstly, the filter mentions in user utterance can be non-standardized (e.g., ram and memory are both surface forms of the filter key ram), implicit (e.g., "hp laptop 16gb" refers to the filter key ram with a value of 16gb) or having complex preferences (e.g., "not windows os", "16gb or more"). user requirements can be ambiguous with no clear mention of the requirement, e.g., "heavy-duty laptop" may imply a highperformance laptop or a rugged one. another layer of complexity arises from the fact that customers may modify their preferences over the course of multiple turns. for instance, a customer utterance such as "what about macbooks with m2" would change their previously stated preference for the cpu from m1 to m2. refer to to address these challenges, we propose a novel architecture for extracting refinements from multiturn conversation history that leverages multiple large language models (llms) as collaborative agents @xcite @xcite . our system (pearl-preference extraction with icl augmentation and retrieval with llm agents) dynamically retrieves the most relevant in-context exemplars from an index during inference time. these icl exemplars @xcite @xcite are a combination of human-curated and synthet- ## problem formulation the conversation between user and chatbot is represented as a sequence of textual utterances u @xmath0, u 1 , • • • u t-1 } where speaker(u j ) ∈ {user, bot} for all @xmath1, t being the current timestamp. we assume that the conversation has been mapped to a product category c for which we have access to the set of valid filter preferences f @xmath2, v i )} containing filter schema keys k @xmath3.g. for "laptop" category, valid filter keys are ram, cpu etc) and set v i of corresponding picker values (e.g. {4gb, 8gb, 16gb, 32gb} for the filter key ram). given the conversation history u t , our goal is to map the user requirement into a set of filter key-value pairs. mathematically, we want to learn a function f such that, refer to figure 1 for an example of multi-turn conversation and user preference extracted by pearl. ## dataset details we analyze conversation logs collected from a chatbot application where human agents acted as shopping assistant to customers for their purchase in a specific product category (laptop). we used chat logs with personally identifiable information (pii) redacted for privacy and compliance. we conduct annotations over complete chats and record the set of preferences of the customer at the end of the chat session. due to the complexity and effort in labelling these chat transcripts, we use a set of ≈ 400 sessions as our test and 100 more labelled chat sessions as our support set for icl exemplars. during the annotation process, we consider preferences of user that can be mapped unambiguously to a combination of filter keys and values @xcite @xcite . refer to appendix a.1 for more details on the internal dataset. to demonstrate generalization of our techniques, we report performance of pearl on the multi-woz @xcite dataset. we specifically choose the 'hotel' domain of multiwoz (multiwoz-h) having the largest number of preference filters, thereby, increasing the complexity of the task. we curate a dataset of ≈ 350 chat sessions as test set and a set of 100 chat sessions as icl exemplars. refer to appendix a.1 for data preparation steps and analyses. ## methodology we propose pearl, a collaborative multi-agent framework that leverages llm-based agents with diverse functionalities to effectively navigate customer preferences in a multi-turn conversational system. note that in our experiments, chat conversations have been pii redacted. figure 1 shows our framework consisting of the primary component for preference prediction accompanied by two auxiliary components: for dynamic retrieval of icl exemplars and icl generation. preference prediction module the preference prediction module is the core component that takes the current utterance from customer and the conversation history as input and generates a dictionary of filter key-values summarizing customer preferences (c.f. figure 1 ). it leverages two llm agents: contextualizationagent and preferencepickeragent. in the first step, contextualizationagent summarizes the entire conversation into a contextualized utterance that captures all preferences of the customer in a singe sentence. subsequently, preferencepickeragent generates a dictionary of filter key-values from the contextualized utterance, filter space f c and icl exemplars. this dual-agent architecture mimics how a salesperson listens to the customer, summarizes their requirement and then executes a search based on those preferences. ## experiments and results to the best of our knowledge, there is no known published work on user preference extraction from multi-turn conversation logs. therefore, we explore strong baselines as mentioned below to evaluate pearl. experiments are performed on llms hosted on aws bedrock and unless specified otherwise all results are reported for claude-instance-v1 (c.f. section 6.2 for results on different choices of llms). refer to appendix a.3 for prompt details. nl2api for this baseline, an llm takes the conversation history and the filter space as input and identifies the preferences as a combination of filters in a single-step prompting approach. three setups are explored: 1) basic (no icl), 2) 10-shot icl with static exemplars, and 3) 10-shot icl 1 an invalid preference pair could be apple branded laptop with nvidia geforce gtx graphics card. with step-by-step chain-of-thoughts reasoning. additionally, nl2api-2s is proposed as a two-step stronger baseline, where the first step determines the filter keys in the conversation, and the second step determines the filter picker values for each key. operatorllm this approach navigates customer preferences through the conversation by iteratively editing the preference set based on each user utterance, using operations like adding, removing, or updating filter key-value pairs. results are reported for three setups: 1) basic (no icl), 2) 10-shot icl and 2) 10-shot icl with chain-ofthoughts reasoning. pearl in our proposed approach, contextualizationagent uses 10 static icl exemplars constructed from real conversation logs and generativeagent augments the initial index of 100 icls with 1.2k synthetic exemplars. for dynamic exemplar retrieval, we use neural representations (refer to ## conclusion in this work, we address a novel challenge of navigating customer preferences in a conversational setting. we proposed pearlwhich is a collaborative multi-agent way of handling this problem. we show that our proposed approach not only outperforms the current production systems, but also has lower latency. we learn that exemplar retrieval and breaking down complex tasks into simpler subtasks during inference is an effective approach to achieve promising results. however, retrieving top-k exemplars might not always be the best idea since diversity and even negative exemplars are helpful to large language models. in future, we would like to study a smarter way of retrieving exemplars during inference to assist our prediction module complete the task with better performance.