I n SCI t: Information-Seeking Conversations with Mixed-Initiative Interactions (2023)

Abstract: In an information-seeking conversation, a user may ask questions that are under-specified or unanswerable. An ideal agent would interact by initiating different response types according to the available knowledge sources. However, most current studies either fail to or artificially incorporate such agent-side initiative. This work presents InSCIt, a dataset for Information-Seeking Conversations with mixed-initiative Interactions. It contains 4.7K user-agent turns from 805 human-human conversations where the agent searches over Wikipedia and either directly answers, asks for clarification, or provides relevant information to address user queries. The data supports two subtasks, evidence passage identification and response generation, as well as a human evaluation protocol to assess model performance. We report results of two systems based on state-of-the-art models of conversational knowledge identification and open-domain question answering. Both systems significantly underperform humans, suggesting ample room for improvement in future studies.1

## introduction recently, there has been increasing interest in developing conversational information-seeking systems @xcite @xcite that assist users in finding information from knowledge sources (e.g., text corpus) via multi-turn conversational interactions. one important advantage of such conversational information-seeking systems is that users do not need to come up with a very descriptive query by themselves @xcite @xcite . in realistic settings, as shown in figure 1 , users can start with a request that is under-specified or has no direct answer, and through conversational interactions, the agent can collaboratively guide users to refine (left) or relax their queries and proactively suggest relevant information that may partially satisfy the user's information needs (right). this collaboration requires a mixed-initiative dialogue, where both the user and agent can direct the flow of the conversation. handling such realistic user requests poses challenges to a conversational agent system. a comprehensive search can result in multiple passages from the knowledge source, which may provide different components of an answer (possibly incomplete) or multiple answers that surface ambiguities in the user query. depending on the available information, the agent needs to use different strategies, which might involve summarizing the results, providing partial information, or trying to clarify an ambiguity. however, existing information-seeking conversation datasets rarely contain conversations where agents initiate different interaction strategies. as a result, most conversational question answering (cqa) work focuses on user-initiative interactions, where the agent simply responds to user questions with direct answers or uses no answer for out-of-scope queries @xcite @xcite . other work studies clarification questions using artificially created data, failing to capture natural information-seeking interactions @xcite @xcite . in addition, most of them only rely on a single evidence string or passage for agent response construction. to support research in mixed-initiative conversations, we introduce inscit (pronounced insight), a dataset for information-seeking conversations with mixed-initiative interactions, where agents take various strategies, such as providing direct answers (72%), raising clarifications (13%), and presenting relevant partial information (13%), to address users' information needs. it contains 805 natural human-human conversations with 4.7k user-agent turns over diverse topics, collected through a scalable annotation pipeline and careful quality control. to simulate realistic information-seeking scenarios, users write queries with minimal restriction, and human agents decide on different strategies to respond, after searching over the knowledge source (i.e., wikipedia) for evidence passages. we formulate two tasks for the conversational agent system: (1) identify a set of evidence passages from wikipedia, and (2) generate a response grounded in the evidence. since handling queries with multiple evidence passages or no direct answer can be open-ended, we emphasize the need for human evaluation, and propose a systematic human evaluation protocol that considers diverse aspects including coherence, factual consistency, and information comprehensiveness. we present two strong baselines based on the state-of-the-art in open-domain question answering @xcite and conversational knowledge identification @xcite . while the systems achieve substantial improvements over a trivial baseline, there is still significant room for improvements, especially for scenarios requiring agent strategies other than providing a direct answer. our analysis suggests that the key remaining challenges are improving passage identification and ambiguous user queries have been observed in single-turn question answering tasks @xcite @xcite , but these are usually addressed by training a model to predict multiple conditional answers without further interaction. a few other studies create artificial conversations to address ambiguous user questions. for instance, qulac @xcite and the data collected in follow-up work @xcite are based on user queries containing a set of pre-specified multi-faceted entities, where agents choose from a fixed set of clarification questions that cover these ambiguities. sharc @xcite , doc2dial @xcite , and multidoc2dial @xcite are rule-based information-seeking conversations in the social welfare domain that incorporate agent-side clarifications. @xcite create abg-coqa by rewriting conversations in the coqa dataset to intentionally include ambiguous questions. in contrast, inscit consists of human-human conversations with natural information-seeking user requests and mixed agent initiative to address them. @xcite crawl conversations from stack exchange foot_1 that are mixed with informationseeking utterances and casual talk. one grounding document is heuristically obtained for each conversation. in contrast, inscit contains validated grounding passages and only goal-oriented agent interactions. knowledge-grounded social chat instead of seeking for information, the user intent in social chat is mostly to conduct casual talk. knowledgegrounded social chat systems @xcite @xcite incorporate external knowledge with the purpose of making the conversations more engaging and informative. @xcite trains a conversational agent to select knowledge to present based on the user's background, in order to maintain the user's interest in the conversation. ## task formulations we define two task formulations for inscit, namely passage identification and response generation. these two tasks mimic how an agent responds to each information-seeking user request, by first searching for relevant information over the knowledge source and then constructing the response based on the gathered information. comparing with prior studies on open-domain information-seeking conversations @xcite , the key challenges in our tasks come from identifying and fusing comprehensive information from multiple passages to construct responses using different strategies, rather than a single passage and a short answer. at the n th agent turn, both tasks have the same input: all previous utterances (i.e., dialogue context) @xmath0, a 1 , u 2 , a 2 , . . . , u n ], the corpus of all passage candidates c, and the previously used passages {p 1 , p 2 , . . . , p n-1 } where each in interactive dialogues, each predicted evidence pi and response āi are used in the dialogue context for later conversations. however, to use pre-collected dialogues with automatic evaluation metrics, the input context must be the same as that leading to the human reference response. this is also consistent with setups in previous information-seeking dialogue studies that are discussed in § 2. therefore, the gold {p i } and {a i } are used here as inputs in testing. ## our data: inscit we introduce inscit, a new information-seeking conversation dataset where the agent interprets the user intent and provides comprehensive information grounded in wikipedia via natural human-human interactions. in this section, we present our data collection pipeline, quality control mechanisms, and analyses that show the characteristics and diversity of the user and agent turns. ## conclusion & discussions in summary, we introduce inscit, a new opendomain information-seeking conversational dataset grounded in wikipedia, with mixed-initiative user-agent interactions. inscit supports two tasks (passage identification and response generation), for which we present results of two strong baselines, with best results obtained with the pipelined dialki+fid system. we also introduce a human evaluation protocol. future work both models significantly underperform humans in both tasks in all metrics. the relative performance gap is greatest for scenarios that require the agent to provide a nondirect answer. we find that passage identification significantly impacts response generation (particularly coherence) by providing relevant grounding knowledge. thus, improving methods for selecting relevant passages is critical for future work. key challenges that remain in response generation are how to fuse and present comprehensive information from multiple passages and learning when and how to use non-direct response strategies. given the small size of our training data, another future direction is to explore transfer learning using existing information-seeking conversation or question answering resources. our work focuses on different strategies that can be adopted by the agent to better address user requests in a conversational question answering setting, assuming the user will either ask an information-seeking question or provide a clarification to the agent. exploring more user-side strategies would be interesting for handling system errors and other types of conversations (e.g., negotiations). in contrast to wikipedia passages, information sources used in practice (e.g., the whole web) can often contain less trustworthy information. in such cases, retrieving evidence passages containing the same answer and predicting the trustworthiness of each answer based on all such retrieved passages can be a promising direction. another direction that is worth future exploration lies in the design of evaluation metrics. we follow previous studies to evaluate the model performance when given a fixed human-human dialogue context. however, as pointed out by @xcite , an interactive dialogue system often needs to handle dialogue contexts containing errors made by the model itself. therefore, it is important for future work to develop new methods for automatic evaluation and scalable human evaluation in the interactive setting.