A Survey on Asking Clarification Questions Datasets in Conversational Systems (2023)

Abstract: The ability to understand a user’s underlying needs is critical for conversational systems, especially with limited input from users in a conversation. Thus, in such a domain, Asking Clarification Questions (ACQs) to reveal users’ true intent from their queries or utterances arise as an essential task. However, it is noticeable that a key limitation of the existing ACQs studies is their incomparability, from inconsistent use of data, distinct experimental setups and evaluation strategies. Therefore, in this paper, to assist the development of ACQs techniques, we comprehensively analyse the current ACQs research status, which offers a detailed comparison of publicly available datasets, and discusses the applied evaluation metrics, joined with benchmarks for multiple ACQs-related tasks. In particular, given a thorough analysis of the ACQs task, we discuss a number of corresponding research directions for the investigation of ACQs as well as the development of conversational systems.

## introduction humans often resort to conversations and asking clarification questions to avoid misunderstandings when collaborating with others. asking clarification questions (acqs) is, therefore, a commonly used mechanism to boost efficiency on humanhuman as well as human-machine collaborative tasks @xcite @xcite . as an example of humanmachine collaboration, conversational systems are developed to not only have a natural conversation with people but also to answer various questions of topics ranging from different domains (e.g., news, movie, and music) in an accurate and efficient manner @xcite . to effectively and efficiently answer various questions, it is essential for many existing conversational systems to capture * equal contribution people's intents. only then can conversational systems accurately reply to a series of questions from users @xcite . nevertheless, one essential issue is that limited research exists on acqs and most systems were trained with inconsistent and limited input of data resources. indeed, in the literature, many studies introduced acqs to assist conversational systems when applying to different / a mixture of domains (e.g., movie @xcite or open domain @xcite ). there is also a lack of commonly agreed benchmark datasets for the development of acqs systems with comparable result analysis. however, on the other hand, in the literature @xcite @xcite , a growing number of studies released publicly available datasets while showing a common interest in the acq research direction. this observed contradiction leads to a necessity for a comprehensive overview of the existing datasets as well as the current status of the acq research direction. by addressing this concern, many growing acqs can be better designed, trained and tested with suitable features from properly selected datasets according to comprehensive guidance. therefore, in this paper, we offer an overview of the current status of the acq research progress. in particular, we aggregate and compare the datasets that have been considered for evaluating recent acq techniques from various aspects, such as their dimension, resource, recency and semantic closeness. afterwards, with the overall discussion of publicly available datasets, we shed light on the model performance while running experiments of corresponding representative techniques on such datasets. note that, we also release our implementation code for such experiments foot_0 . next, we summarised the concluding remarks as well as followup suggestions for developing the acq techniques. ## conversational systems a conversational system functions to assist users while addressing various tasks or acting as a partner in casual conversations @xcite . in particular, conversation systems can be classified into four main categories: (1) conversational search (conv. search); (2) conversational question answering (conv. qa); (3) task-oriented dialogues systems (tdss); and (4) social chatbots @xcite . in particular, the first two types, conv. search and conv. qa, extend the classic search and qa systems to a conversational nature @xcite . for tdss and social chatbots, they are more recent research topics and were introduced to build systems for assisting users while addressing a specific task or offering emotional connection and companionship via conversations @xcite . however, due to the limited resources that investigate the challenge of asking clarification questions when developing these two systems, this study focuses on conv. search and conv. qa systems. moreover, acqs in conversational systems partially focus on three main tasks, namely, clarification need prediction (t 1 ), asking clarification questions (t 2 ), and user satisfaction with cqs (t 3 ) @xcite @xcite . first, t 1 evaluates the necessity of asking clarification questions when users provide their initial queries or requests. next, with a positive decision, we turn to the action of providing suitable clarification questions (i.e., t 2 ) by following two main routines: generation or selection from a pool of candidate clarification questions. afterwards, the third task t 3 is to evaluate the effectiveness of the corresponding clarification questions while considering user satisfaction levels from multiple aspects (e.g., the usefulness or relevance of clarification questions). an effective acq-encoded conversational system requires a joint effort to address the three tasks satisfactorily to enhance users' conversational experience. therefore, in this survey, we explore the relevant acq datasets and discuss their suitability while addressing the above three tasks. ## acq datasets in this section, we describe the main characteristics of the existing and relevant acq datasets. note that we include some additional information, such as the corresponding institution, in appendix a. a careful dataset selection and aggregation strat- ## evaluation metrics in this section, we detail the description of the applicable evaluation metrics for the included datasets when evaluating acqs approaches. in particular, as previously discussed, we discuss such metrics accordingly if they are automatic or human-involved. ## discussion and future challenges from the exploration of datasets as well as the experimental results on them, in this section, we highlight the concluding remarks on the current status of the acq research task, mainly from the dataset point of view. in addition, we discuss the promising directions based on the main findings listed below. in the literature, only the mimics dataset was collected by using a clarification pane that simulates such interactions. this makes it challenging to evaluate models in a near-realistic scenario and to estimate how well they could perform in a real-world setting. (3) inconsistent dataset collection and formatting. many included datasets in this paper are frequently presented in distinct structures and can only be applied with a tailored setup. this is a problem while developing techniques and evaluating them on multiple datasets. (4) inconsistent model evaluation. many newly introduced models apply customised evaluation strategies even while using an identical dataset for addressing a specific asking clarification task. this lead to difficulties in model performance comparison.