Conversational Search with Mixed-Initiative - Asking Good Clarification Questions backed-up by Passage Retrieval (2022)

Abstract: We deal with the scenario of conversational search, where user queries are under-specified or ambiguous. This calls for a mixed-initiative setup. User-asks (queries) and system-answers, as well as system-asks (clarification questions) and user response, in order to clarify her information needs. We focus on the task of selecting the next clarification question, given conversation context. Our method leverages passage retrieval from background content to fine-tune two deep-learning models for ranking candidate clarification questions. We evaluated our method on two different use-cases. The first is an open domain conversational search in a large web collection. The second is a task-oriented customer-support setup. We show that our method performs well on both use-cases.

## introduction a key task in information and knowledge discovery is the retrieval of relevant information given the user's information need (usually expressed by a query). with the abundance of textual knowledge sources and their diversity, it becomes more and more difficult for users, even expert ones, to query such sources and obtain valuable insights. thus, users need to go beyond the traditional ad-hoc (one-shot) retrieval paradigm. this requires to support the new paradigm of conversational search -a sophisticated combination of various mechanisms for exploratory search, interactive ir, and response generation. in particular, the conversational paradigm can support mixed-initiative: namely, the traditional user asks -system answers interaction in addition to system-asks (clarification questions) and user-answers, to better guide the system and reach the information needed @xcite . existing approaches for asking clarification questions include selection or generation. in the selection approach, the system selects clarification questions from a pool of pre-determined questions @xcite . in the generation approach, the system generates clarification questions using rules or using neural generative models @xcite . in this work we focus on the selection task. while the latter (i.e., generation) may represent a more realistic use-case, still there is an interest in the former (i.e., selection) as evident by the clarifying questions for open-domain dialogue systems (clariq) challenge @xcite . moreover, the selection task represents a controlled and less noisy scenario, where the pool of clarifications can be mined from e.g., query logs. in this paper we deal with content-grounded conversations. thus, a conversation starts with an initial user query, continues with several rounds of conversation utterances (0 or more), and finally ends with one or more documents being returned to the user. some of the agent utterances are marked as clarification questions. the task at hand is defined as follows. given a conversation context up to (and not including) a clarification-question utterance, predict the next clarification question. a more formal definition is given in section 3.2 below. intuitively, clarification questions should be used to distinguish between several possible intents of the user. we approximate those possible intents through passages that are retrieved from a given corpus of documents. a motivating example from the @xcite challenge is given in figure 1 . the user wants to get information about the topic all men are created equal. through the retrieved passage, the system can ask the mentioned clarification questions. we use two deep-learning models. the first one learns an association between conversation context and clarification questions. the second learns an association between conversation context, candidate passages and clarification questions. evaluation was done on two different use-cases. the first one is an open domain search in a large web corpus @xcite . the second is an internal task-oriented customer-support setup, where users ask technical questions. we show that our method performs well on both use-cases. corpus we focus on works that deal with clarificationquestions selection. @xcite describes a setup very similar to ours for the aforementioned task. they apply a two-step process. in the first step, they use bert @xcite to retrieve candidate clarification questions and, in the second step, they re-rank the candidates using multiple sources of information. among them are the scores of retrieved documents using the clarification questions. however, they do not look at passage content as we do. the clariq foot_0 challenge organized a competition for selecting the best clarification questions in an open-domain conversational search. the system by ntes along (ou and lin, 2020) was ranked first. they first retrieve candidate clarification questions and then re-rank them using a roberta @xcite model, that is fine-tuned on the relation between a query and a clarification question. unlike our method, they do not exploit passage content. in rao and daumé iii (2018), they select clarification questions using the expected value of perfect information, namely a good question is one whose expected answer will be useful. they do not assume a background corpus of documents. ## conclusions we presented a method for clarification-questions selection in conversational-search scenarios that end with documents as answers. we showed that using passages, combined with deep-learning models, improves the quality of the selected clarification questions. we evaluated our method on two diversified dataset. on both datasets, the usage of passages for clarificationquestions re-ranking achieved improvement of 12% -87% over base ir retrieval.