P ro MIS e: A Proactive Multi-turn Dialogue Dataset for Information-seeking Intent Resolution (2024)

Abstract: Users of AI-based virtual assistants and search systems encounter challenges in articulating their intents while seeking information on unfamiliar topics, possibly due to complexity of the user’s intent or the lack of meta-information on the topic. We posit that an iterative suggested question-answering (SQA) conversation can improve the trade-off between the satisfaction of the user’s intent while keeping the information exchange natural and cognitive load of the interaction minimal on the users. In this paper, we evaluate a novel setting ProMISe by means of a sequence of interactions between a user, having a predefined information-seeking intent, and an agent that generates a set of SQA pairs at each step to aid the user to get closer to their intent. We simulate this two-player setting to create a multi-turn conversational dataset of SQAs and user choices (1025 dialogues comprising 4453 turns and 17812 SQAs) using human-feedback, chain-of-thought prompting and web-retrieval augmented large language models. We evaluate the quality of the SQs in the dataset on attributes such as diversity, specificity, grounding, etc, and benchmark the performance of different language models for the task of replicating user behavior.

## introduction users of ai-based virtual assistants and search systems such as google search, alexa, bing, etc. often face challenges in effectively satisfying their information-seeking intents, especially on unfamiliar topics. this stems from a combination of (i) the inability of the user to formulate the appropriate question(s) for the agent owing to the complexity of the intent, (ii) the user lacking meta-information on an unfamiliar topic that is required to phrase the appropriate question(s) to the agent, and (iii) the agent's response being long, complicated and cognitively challenging for the user to process. ## related work proactive conversational systems several research studies have explored the topic of clarification question generation @xcite and question disambiguation @xcite . aliannejadi et al. (2021) proposed the clariq dataset of open domain dialogue for predicting and generating clarification questions. guo et al. (2021) and deng et al. (2022) propose datasets (abg-coqa and pacific respectively) in this domain for disambiguity prediction, clarification question generation and conversational qa. zhang et al. (2018) proposed the proactive 'system ask user respond' setting for improving conversational search. @xcite @xcite acquire user preference through multiple turns of interactions using rl-based conversational recommendation systems. these works, however, are constrained to the product domain and only focus on one feature per turn. @xcite propose a keyword-guided conversational model for reaching a target keyword. our work extends this by enhancing the complexity of user intent from keywords to open-domain natural language constructs. @xcite propose a rl-based approach for generating informationseeking questions starting from short initial user queries. however, this approach is restricted to single-turn sq generation, and does not contain answers to the generated sqs. seeker @xcite highlights that search and knowledge augmented dialogue outperforms previous state-of-theart models in open-domain knowledge-grounded conversations on aspects of consistency, knowledge and per-turn engagement. llms and dialogue large language models (llms) have shown state-of-the-art reasoning abilities, along with zero-shot and few-shot generalization capabilities @xcite . internet-augmented dialogue generation @xcite proposes an approach to generate a web search query based on the dialogue and using the search results to condition the llm's output. @xcite propose multi-stage prompting for knowledgeable dialogue generation that increases knowledge, relevance and engagement without fine-tuning the model. @xcite propose the proactive chain-of-thought prompting scheme to augment llms with goal planning and generating clarification questions. @xcite use in-context learning to generate diverse questions in task oriented dialogues based on user goals. @xcite use llms for planning and reasoning to provide a more personalized and engaging experience for the user query. ## the promise setting we first formally define the proactive multi-turn dialogue for information-seeking intent resolution setting. consider an interaction between a user u and an ai-agent a. the user u has an informationseeking intent i. based on meta-information that the user has on the topic of i, the user formulates an initial question q 0 to ask a to initiate the information-seeking dialogue. at each turn i, the agent a uses the conversation history with u to create a set of l suggested questions (sqs) s i : {s i 1 , s i 2 , . . . , s i l } that may be relevant for the user. the user then chooses sq s i u from the set s i of sqs created by a in turn i, or indicates that none of s i are relevant to their intent. after making the choice, a provides the answer to s i u to u. at the end of each turn, u indicates if their original information-seeking intent i has been satisfied or if they still need more information on some aspects of i. the conversation continues till the user signals that their information-seeking intent has been satisfied. we illustrate the promise setting in fig 2 . we describe information available to u and a below: agent: at each turn i of the conversation, the agent a has access to the conversation history with the user including the initial question q 0 , and previously generated sqs and choices made by the user: note that a does not have access to the information-seeking intent i of the user. user: at each turn i of the conversation, the user u makes a choice s i u from the set s i of sqs created by a using the previous conversation history with the agent including: the initial question q 0 , previously generated sqs and choices made by the user: {s 1 , s 1 u }, {s 2 , s 2 u }, . . . , {s i-1 , s i-1 u } and the information-seeking intent i. ## the promise dataset to curate the dataset, we implement a two-player setting as shown in fig 2 where one player acts as agent while the other player acts as user. we use a web-retrieval augmented language model as the agent. we now describe our methodology for simulating the agent and the user below: ## simulating human users using llms using the collected dataset, we want to study how effectively can language models mimic the reasoning of users (humans) in carrying forward an information-seeking exchange with an agent to satisfy an intent. simulating users effectively can improve the velocity of collection of dialogue datasets and facilitate privacy-aware evaluations. the problem of simulating the user can be split into two tasks (statistics in we observe that some models like dolly-v2-7b and mpt-7b-story are unable to effectively follow instructions and end up generating 'intent not satisfied' for a majority of samples (thereby obtaining imbalanced f1 scores for the two classes). the qlora fine-tuned llama-7b and vicuna-7b perform significantly better than their few- we notice that none of the models are able to achieve very high macro-f1 scores for either of the two tasks (task 2 having significantly lower macro-f1 scores than task 1). this highlights a big performance gap in the performance of state-ofthe-art llms with humans for this task of resolving information-seeking user intents. given how fundamental this task is for virtual assistants and search engines, we believe that our promise dataset will help encourage research on this problem and improve performance of llms on this task. ## conclusion we introduce a new setting: promise aimed at improving ai-based virtual assistants and search systems to resolve information-seeking user intents in an end-to-end manner. we create and release a dataset of high-quality conversational data collected using human annotations and llms. we analyze the quality of the dataset and benchmark the performance of popular llms as user-simulators. the promise framework and dataset will be beneficial in enhancing intelligent systems' user experience by making it interactive and proactive.