Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions (2021)

Abstract: Enabling open-domain dialogue systems to ask clarifying questions when appropriate is an important direction for improving the quality of the system response. Namely, for cases when a user request is not specific enough for a conversation system to provide an answer right away, it is desirable to ask a clarifying question to increase the chances of retrieving a satisfying answer. To address the problem of ‘asking clarifying questions in open-domain dialogues’: (1) we collect and release a new dataset focused on open-domain single- and multi-turn conversations, (2) we benchmark several state-of-the-art neural baselines, and (3) we propose a pipeline consisting of offline and online steps for evaluating the quality of clarifying questions in various dialogues. These contributions are suitable as a foundation for further research.

## introduction the ultimate goal of a conversational system is to assist users by returning an appropriate answer in response to their requests @xcite . recent progress on neural approaches to natural language processing @xcite @xcite , and the availability of large amounts of conversational data have triggered a renaissance in end-to-end neural open-domain chatbots @xcite @xcite @xcite . there has been great progress on suggesting measures to evaluate what makes a conversation satisfying for users using various human evaluation techniques @xcite . those efforts showed that suggested large pre-trained models do not always perform seamlessly @xcite , and there are still several challenges needed to be solved for open-domain conversational systems @xcite . are you interested in a specific type of product? yes, surface treatment products that jac chemical company manufactures here is information about their products: (b) and (c) demonstrate a situation when the request is ambiguous and a system needs to act. @xcite conclude that people have similar expectations from talking to bots and humans. this similarity is a possible explanation for why sometimes user requests might be ambiguous and incomplete, as shown in fig. 1 (b) and (c ). this ambiguity is especially challenging to handle in a dialogue setting, where a system is limited by returning only one answer in response to each request, unlike in web search setup where diversification of results is possible and acceptable @xcite . previous research has shown that users are much more forgiving about system mistakes if they can act on them with minimal efforts spent @xcite . therefore it is more appropriate to ask a clarifying question in user request ambiguity rather than generating incorrect answers. there are separate attempts to explore the following related tasks: (1) identifying a moment when the question should be asked in the course of conversation @xcite ; and (2) retrieving a clarification question @xcite . in this paper, we aim to combine these related aspects and study the following problem of generating clarifying questions for open-domain conversations: the system must identify wherever the question is ambiguous, and, if so then instead of trying to answer it directly, it should ask a good clarifying question (fig. 1 ). one possible stumbling block preventing the community from studying the problem of open-domain clarifying question generation to enhance user experience while interacting with a conversational bot @xcite is the lack of suitable datasets, which we address in this work. to summarise, the main contributions of this work are: c1 releasing a dataset dedicated to the problem of asking a clarifying question in open-domain dialogue systems. the dataset includes single-(∼15k) and multi-turn (∼1.5m) conversations and covers ∼300 various topics and it is suited to study: (1) when a clarifying question should be asked given the current context of the conversation; and (2) which question should be asked; c2 benchmarking several state-of-the-art (sota) ## related work our work is broadly relevant to two strands of research: learning to ask clarifying questions in opendomain conversational settings (section 2.1) and evaluating dialogue systems (section 2.2). ## problem setting our main goal is to collect a dataset to enable studying clarifying questions generation whenever appropriate, as depicted in examples in fig. 1 . fig. 2 demonstrates a pipeline that makes it possible to process user requests in the open domain as follows: 'user request understanding' (uru) decides which module to call either 'clarifying question generation' (cqg) or 'answer generation' (ag). in this work, we focus on the first two. we aim to collect the following data: • user request (u ): an initial user request in the conversational form, e.g. the collected dataset of pairs (u, {q, a}) can be easily transformed to a set of single-turn conversations consisting of the coherent and consistent triples (u, q, a) as shown in the example in fig. 2 . we ask items in a triple, u , q and a, satisfy the following requirements: r1 user requests (u ) must cover various conversational topics to represent open-domain dialogues; r2 the final collection of u should contain both types: ambiguous and unambiguous; r3 each inquiry u to the system should be in the conversational form; r4 the need for clarification should be predetermined as a label for each u in the collection; r5 each clarifying question (q) should be reasonable, coherent with u and address multiple facets of every ambiguous request u ; and r6 each user answer a should be consistent with the clarifying question from the system. after collecting of single-turn conversations, they are used to train various conversational agents. to collect multi-turn conversations, the two bestperforming agents are utilized to converse with crowdsourced workers, who evaluate a system quality and reply to suggested clarifying questions. finally, the two agents are evaluated using acute-eval framework @xcite , which is best available practice for online evaluation of open-domain dialogue systems. overall, our pipeline for data collection and evaluation is summarized in fig. 3 . ## data collection following the suggested pipeline in fig. 3 this section describes our results regarding the collected datasets to initiate more follow-up studies of the problem of 'asking clarifying questions in opendomain dialogues', namely, p 1 : single-turn dialogues (sec. 4.1) and p 3 multi-turn ones (sec. 4.2). ## models and evaluation following the suggested pipeline in fig. 3 , we explain our contributions regarding the evaluation of 'asking clarifying questions in open-domain dialogues' problem, namely: • p 2 : how the dialogue agents are trained and automatically evaluated based on single-turn conversations (sec. 5.1); • p 4 : how evaluation of multi-turn conversations is performed from both perspectives: offline automatic manner and having human-in-the-loop using acute-eval @xcite (sec. 5.2). we design our experiments to collect answers to the following research questions: rq1 when to ask clarifying questions during open-domain dialogues? (sec. 5.1.1) rq2 which clarifying question to ask for a given context of a conversation? (a. the single-turn conversations case is described in sec. 5.1.2; b. multi-turn one -sec. 5.2) ## conclusions asking clarifying questions when a user request is ambiguous is essential for developing human-like open-domain dialogue systems. in this work, we introduce a large-scale dataset that covers almost 300 different topics and is suitable as a foundation for further research. the collected dataset includes both single-and multi-turn conversations. we benchmark several state-of-the-art neural models that were fine-tuned for asking clarifying questions. based on how these models performed, we conclude that they are solid baselines for future research that more fully explores the problem space. in this paper, we also suggest an offline automatic evaluation pipeline, which agrees with human-inloop evaluation. we publicly release the collected datasets, the code for training baselines, and our evaluation procedure in order to push forward the state-of-the-art.