Redefining Proactivity for Information Seeking Dialogue (2024)

Abstract: Humans pay careful attention to the interlocutor’s internal state in dialogues. For example, in recommendation dialogues, we make recommendations while estimating the seeker’s internal state, such as his/her level of knowledge and interest. Since there are no existing annotated resources for the analysis and experiment, we constructed RecomMind, a movie recommendation dialogue dataset with annotations of the seeker’s internal state at the entity level. Each entity has a first-person label annotated by the seeker and a second-person label annotated by the recommender. Our analysis based on RecomMind reveals that the success of recommendations is enhanced when recommenders mention entities that seekers do not know but are interested in. We also propose a response generation framework that explicitly considers the seeker’s internal state, utilizing the chain-of-thought prompting. The human evaluation results show that our proposed method outperforms the baseline method in both consistency and the success of recommendations.

## introduction generally, the aim of information-seeking dialogue (isd) agents @xcite is to generate an informative response which answers the user's query. in these interactions, users typically pose questions to obtain specific pieces of information, and the dialogue agent generates coherent responses which contains the information requested by the user. in recent years, large language models (llms) have generally succeeded at achieving this goal @xcite . however, current isd agents, as well as llms in general, tend to be more reactive than proactive. an example of a reactive * work done during internship at amazon agi. response is provided in figure 1 . responses generated by a reactive isd agent would adequately address the user's query but fail to proactively engage the user. once the requested information is provided, the conversation with the isd agent naturally concludes. in isd, existing work on proactivity primarily focuses on generating clarifying questions and eliciting user preferences @xcite , aiming to resolve ambiguity in the user's query or uncover their preference respectively. current definitions of proactivity in isd do not emphasize engaging the user or sustaining the conversation once the desired information has been provided. hence, we introduce a novel definition of isd proactivity that emphasizes generating responses that aim to sustain the interaction by proactively engaging the user via the introduction of new information pertinent to the initial query. by proactively providing new related information, the agent can stimulate the user's interest, prompting further inquiries and sustaining the conversation. hence, our definition of isd proactivity focuses on actively delivering information related to the initial query in a conversational manner, thereby naturally guiding the conversation towards addressing multiple pieces of information, improving the overall informativeness during interactions with users and further enhancing user satisfaction @xcite . unlike prior definitions, we focus on the proactiveness of each individual response, evaluating them individually rather than as part of the entire conversation. this allows us to evaluate responses on specific criterion (section 3). according to our definition, a proactive response consists of the answer to the user's query and a proactive element, which refers to new information related to the initial query. the proactive element can be further categorized as either a follow-up question (fq) or additional information(ai). samples of proactive responses according to our definitions are also provided in figure 1 . it's important to note that this work does not encompass factual accuracy or information correctness. the focus is purely on syntactic and semantic proactivity. in this paper, our contributions are as follows: proactive dialogue proactive dialogue encompasses various techniques for engaging users by steering conversations in specific directions. in the context of open-domain (od) dialogue, some popular proactive dialogue tasks include: target-guided dialogue, prosocial dialogue, and non-collaborative dialogue. target guided dialogue focuses on directing interactions toward predefined topics or entities, using methods such as response planning @xcite , event-based knowledge graphs @xcite , and commonsense bridging @xcite . prosocial dialogue involves generating non-offensive responses that adhere to societal norms @xcite . in the context of task-oriented (to) dialogue, proactive dialogue definitions include non-collaborative dialogue as well as enriched to dialogue. in noncollaborative dialogue, the agent and user have opposing objectives. some examples include persuasion @xcite , negotiation @xcite , and deception-based dialogue @xcite . enriched to dialogue shares some similarities with our task. however, while enriched to dialogue focuses on enhancing conversational naturalness through additional information, our goal is to sustain isd. rather than prioritizing naturalness, we aim to encourage user engagement by introducing new information (either directly or through a fq) that prompts the user to continue the conversation. with regard to isd specifically, response proactivity largely revolves around generating clarifying questions and eliciting user preferences @xcite . clarifying question generation aims to resolve ambiguity in user queries to provide the user with the requested information @xcite . approaches include retrieval and ranking-based frameworks @xcite , reinforcement learning with clarification utility rewards @xcite , and multistep frameworks predicting the need for a clarifying question before generating one @xcite . some methods also combine clarifying questions and conversational qa in multi-turn context @xcite . user preference elicitation involves proactively reveals the user's interests for better recommendations @xcite . this task is often treated as a decision-making problem often tackled with reinforcement learning @xcite @xcite . unlike earlier definitions, we do not concentrate on specific proactive isd aspects like clarifying question generation or user preference elicitation. instead, we solely focus on enhancing proactivity by providing relevant information. moreover, we evaluate the proactiveness of each individual response separately, rather than considering the entire conversation. llm-based isd in recent years, llms have emerged as leading models in language generation tasks, demonstrating state-of-the-art performance. in isd, recent methods utilize llms through incontext learning or supervised fine-tuning. incontext learning refers to learning a new task during inference with a few prompt examples. approaches leveraging few-shot @xcite and @xcite prompts have been employed in this context. llms are also often trained on dialogue contexts alongside task instructions, which is known as instruction tuning, to enhance zero-shot performance. in the context of dialogue, llms such as flan-t5 @xcite , instructgpt @xcite , and instructdial @xcite were explicitly trained on dialogue data for chat applications. likewise, instruction-tuning has also been applied to improve the accuracy and informativeness of conversational qa responses @xcite . these methods excel at achieving the primary aim of isd to address user queries. however, as highlighted in section 1, they tend to produce reactive responses that do not proactively engage the user. ## problem definition we propose a new proactive response definition for isd that consists of two components: an an-swer and a proactive element. the answer directly addresses the user's query, while the proactive element actively engages the user by providing related information. the proactive element enriches the user's understanding and can spark further interest, prompting them to further engage the conversation to find out more. we further classify the proactive element into two main categories: additional information (ai) and follow-up questions (fqs). ai refers to any knowledge not explicitly requested in the user's query or mentioned in the answer, but that could be of interest to the user. the provision of high-quality ai enriches the conversation by increasing its informativeness, and encouraging the user to continue the interaction. to determine if an ai qualifies, the following criteria must be met: it's important to note that llms often have a tendency to include excessive details in a single response, which can hinder naturalness, particularly in spoken context. our goal is to incorporate ai in a concise and engaging manner that encourages the user to continue the interaction. a fq asks if the user is interested in a specific piece of additional relevant information related to their initial query. the information itself is not explicitly provided in the fq. by asking appropriate fqs, we can extend the conversation beyond the initial turn. the criteria for a fq are defined as follows: figure 2 presents examples of responses that do not meet the previously mentioned criteria. unlike prior work in isd, our definition focuses specifically on response proactivity rather than factual accuracy. therefore, we do not include criteria related to information accuracy or ground responses on external knowledge sources. there factors are often used to prevent hallucination and ensure factual correctness. ## proactive response evaluation in this section, we propose several automatic metrics to quantify the proactivity of a response. a reliable automatic metric would enable objective and cost-effective evaluation, ultimately enhancing the reproducibility of our work. ## corpus construction to create our proactive dialogue corpus, we utilize the natural questions question answer (nqqa) dataset @xcite . each sample in this dataset includes a query, a short answer, and a long answer. the short answer provides the response to the query, while the long answer contains some relevant information. we selected the nqqa corpus because the query and short answer format resembles a typical single-turn conversation between a human and an isd agent. however, since the short answer in the nqqa corpus consists of only a single entity, it needed to be modified for conversational naturalness. ## proactive response generation in this section, we describe the in-context learning and instruction-tuning approaches we employed for proactive response generation. ## experiments instruction tuningimplementation in our experiments, we utilize the 40b instruction-tuned falcon llm @xcite and the 13b stablevicuna llm @xcite . results attained using stablevicuna are provided in the appendix a.2. we utilize a temperature value of 0.2 for all generations. for each proactive element, we split our proactive dialogue corpus into two distinct sets: a 500-sample training set and a 500-sample test set. we select demonstration examples for our prompts from the training sets, and then evaluate them on the test set. we instruction-tune the llm on the training sets for both the fqs and ai concurrently. the instructions used are identical to the direct prompt. metric correlations the prompt-based baseline yields low correlation scores, highlighting its limitations as a metric. conversely, the classification-based baseline achieve better, though inconsistent, correlations with human evaluations. specifically, correlations for ai are higher than those for fqs. this difference arises because negative samples for ai, which mostly violate the informativeness criteria, are simpler for the model to detect compared to the nuanced, generic responses that characterize negative samples for fqs, which violate the specificity criteria. future research could involve improving the correlations through further prompt engineering or by enriching the training dataset with more varied negative examples. the proposed semantic and sentiment scores clearly outperform both baselines. the semantic metric, encompassing relevance, informativeness (ai), and specificity (fqs), achieves the highest correlation scores. this aligns with expectations, as many negative responses lack the required informativeness and relevance. conversely, the sentiment score focuses on perspective and conversational naturalness, which are less common in negative samples. therefore, we recommend using both metrics together to effectively evaluate response proactiveness, covering the criteria outlined in section 3 comprehensively. in-context learning scores attained by the direct, 3-step cot, and 3-in-1 cot prompts on falcon-40b-instruct are shown in it is also evident that the 3-step cot prompt surpasses both the 3-in-1 cot and direct prompts when it comes to the fq. conversely, for ai, the 3-in-1 cot prompt outperforms both the 3-step cot and direct prompts. this could be attributed to the inherent difficulty in generating high-quality fqs for the llm, which generally excels at generating informative responses. consequently, the fq task benefits more from the 3-step cot prompt since it breaks down the task into three simpler components. instruction tuning ## multi-turn setting to demonstrate the efficacy of our approach in the multi-turn conversations, we sampled 50 test cases from our dataset and interactions between a simulated user and an agent using falcon-40b-instruct. we used 3-step and 3-in-1 cot prompts with modifications to produce proactive responses, detailed in appendix a.8. after conducting 50 simulations, we discovered that when the agent includes ai or fq, the user is significantly more inclined to continue interacting with the agent. in contrast, responses lacking this proactive element usually consist of the agent merely acknowledging the information provided, naturally ending the conversation ( ## conclusion in this work, we propose a novel response-level definition of isd proactivity. per our definition, a proactive response includes both an answer and a proactive element (fq or ai). we compiled a dataset consisting of 2000 single-turn dialogues, and introduced a novel 3-step cot and 3-in-1 cot prompt that outperforms standard few-shot prompts in generating proactive responses. future work could entail exploring finer-grained proactive elements or employing reward modelling and reinforcement learning with human feedback (rlhf) for fine-tuning. expanding the current corpus to the multi-turn scenarios could also facilitate further research to improve in-context learning or supervised fine-tuning performance. existing conversationlevel metrics in isd could also be enhanced to account for response-level proactivity. the performance of different llms on our task can also be explored.